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I. INTRODUCTION 


A. AREA OF RESEARCH 

The researcher will identify and examine the elements of 
quality in the contractor performance of Department of Defense 
O). contracts. Once „identified, the most salient and 
quantifiable quality factors will be incorporated into a model 
source Selection Evaluation Procedure for use by Procurement 
Contracting Officers (PCO’s) and other source selection 
personnel involved in formal source selection. The researcher 
will then examine various sources of contractor present and 
past performance and a recommended Performance Risk Assessment 
MENA procedure from the Army, and incorporate it into a 
procedure for Source Selection Evaluation Boards (SSEB’s) or 
source Selection Authorities (SSA's) to verify and validate 
@erereod contractor quality information with actual contractor 


performance quality history. 


B. RESEARCH QUESTIONS 
1. Primary Research Question 
What would be a useful and practical Source Selection 
Evaluation Procedure model for PCO's and SSEB’s which 
Merc incorporate significant and quantifiable quality factors 
Aen a procedure to verify and validate offered contractor 


Seality information with contractor quality history? 


2. Subsidiary Research Questions 

1. How are Packard Commission Ete unces and 
recommendations and the DOD-wide TOM implementation ' 
efforts spawning attempts ae incorporate qualit, 
considerations into DoD source selection evaluation methods? 

2. How did CICA impact DoD use of Best Value source 
selection procedures? 

3. What are the current attitudes of Navy PCO’s, PM’s 
and commercial procurement personnel towards quality factors 
in source selection procedures? 

4. Which quality factors are considered as the most 
critical and quantifiable to Navy PCO's today? 

5. What are the various systems throughout DOD which 
collect and disseminate historical contractor quality history 
data, and can be utilized in the conduct of a Performance Risk 


Assessment (PRA)? 


C. OBJECTIVE 

The objective of this research is to propose a source 
selection method which addresses one of the recommendations of 
the Packard Commission as well as to possibly aid in the 
implementation of Total Quality” Management am) DoD wae 
constructing a model Source Selection Evaluation Procedure 
which can add quality factors and TOM to existing technica 
and cost considerations. The information which the model 


quality factors request in solicitations wil Uii thene 


validated and verified by the SSEB against contractor quality 
history as reflected in centralized past performance data 
collection bases and other sources of past performance 
quality. Therefore, if several contractors are relatively 
Seoul or close im technical abilities and “cost, the quality 
factor could be a valid tie breaker, depending on how this 
model is used or how much weight is placed on quality 


Considerations. 


D. SCOPE OF THE THESIS 

This study will be specifically limited to procurement 
eerlons oi such magnitude or importance as to require formal 
source selection procedures, with an emphasis on the 
acquisition of major weapons systems on a greatest or best 
value basis. The background research will concentrate on the 
period 1984-1991, the period just after the implementation of 
CICA to the present. The quality factors, especially TQM, will 
Be. drawn irom the application guidelines for the Malcolm 
Baldridge National Quality Award. The most current and 
comprehensive service/agency initiatives for collecting and 
Gisseminating contractor quality information will also be 
examined as part of the Performance Risk Analysis portion of 


the model. 


E. METHODOLOGY 

This study will consist of three phases. The first phase 
will combine extensive literature review and research with 
personal interviews from appropriate DOD and private sector 
personnel. This phase will be finished upon completion of a 
model Source Selection Evaluation Procedure, and procedures to 
verify the offered contractor information. 

The second phase will consist of the distribution o en 
model and procedures to selected DoD and private sector 
personnel for critical review and comment. The selection 
process will not be random, stratified or in any way 
scientifically structured, but will be centered "amu 
selection of knowledgeable, experienced individuals who can 
objectively review and analyze the model for possible 
improvements or refinements. 

The third phase will consist of adjusting or moditying@ee— 
model and procedures with valid inputs from the selected 
experts. The resulting product should be more flexible and 
adaptable for possible use by PCO’s and SSEB’s in DoD 


contracting activite: 


F. ORGANIZATION 
The remainder of the thesis: is organized into the 
following chapters: 


Le Chapter II will provide background information on 
and a description of quality, on what Total 


Quality Management is, what the current attitude 
towards quality is in the Navy as a source 
selection evaluation factor in procurements, how 
the FAR addresses quality in source selection, and 
what are the primary forces influencing quality 
perceptions ana utilization” in DOD since CICe, A 
description of the Army’s use of TOM and other 
continuous process improvement processes as 
evaluation and performance risk assessment factors 
in an actual major procurement will also be 
presented. 


Chapter III will be a model Source Selection 
Evaluation Procedure integrating TOM and past 
performance into a feasible method for DoD 
Somtraccing aACEL VICES: 


Chapter IV will examine performance risk, will 
address the legal aspects of utilizing past 
performance data in performance risk, will 
describe the Performance Risk Assessment procedure 
as it is generally conducted in DoD, and will list 
and describe some of the various sources of past 
performance data available to DoD procurement 
CCa lS today: 


Chapter V will synopsize any feedback received 
from reviews and critiques of the model plan, and 
will contain any revisions to the model based upon 
this feedback. 


Chapter VI will provide conclusions based upon 
findings and recommendations regarding the 
feasibility- -of Utilizing the modeldprocedure and 
ehes future of quality “cCactors “in DoD! source 
selection. Areas of study that warrant further 
research will also be identified. 


II. BACKGROUND 


D: INTRODUCTION 

This chapter will provide background information on what 
quality is perceived to be, what Total Quality Management is, 
what the current Navy attitude is towards quality as a source 
selection evaluation factor in procurements, and how the 
Federal Acquisition Regulation addresses quality in source 
selections. A general overview of the primary forces 
influencing attitudes toward quality in DoD source selection 
procedures since CICA implementation, and of the Army’s 
utilization of quality factors in an actual procurement wa 


also be presented. 


B. WHAT IS QUALITY? 

Quality in products or services iS not as easily 
quantifiable or describable as other characteristics such as 
dimensions, weight, performance parameters or cost/price. In 
essence, the relative elusiveness of a concrete definition for 
use in solicitations or of an accepted method to validate 
offeror quality claims against actual past performances 
made quality a seldom used evaluation factor for DoD PCO's. 
Quality has been a heavily used factor in commercial/industry 


purchasing offices for many years, and there exists a 


significant difference in perceptions of quality between the 
Government PCO's and private purchasing officials. 

When searching for a commonly accepted definition of 
aquel y,. the writings oí four prominent authors in the field 
of quality management: Crosby, Juran, Deming and Garvin, 
Stand out as being the most comprehensive and widely accepted. 

Pee: “rosby adetanes “quality as. simply, "conformance to 
requirements" [Ref. 1: p. 14]. His main point regarding 
quality is that it provides cost effectiveness and value as a 
product characteristic because the costs of scrap, rework, 
service, warranty, inspections and tests which result from 
noncomformance to requirements or specifications far exceed 
the cost of efforts to design and produce products or services 
Heme OO not. fail in the field” [Ref 1: p. 15-16]. 

J.M. Juran’s simple definition of quality is "fitness for 
use" [Ref. 2: p. 2-2]. The user or customer makes this fitness 
determination based upon features of the product/service the 
user recognizes as beneficial. 

W. Edwards Deming is arguably the most widely known and 
respected author and practitioner in the field of quality 
management, in fact his name is synonymous with Total Quality 
Management (TOM). He is credited for successfully implementing 
the total quality approach and culture into Japanese 
manufacturing. He does not offer a firm working definition of 
quality, rather he describes the difficulty of defining 


euality [Ref. 3: p. 169]: 


The difficulty in defining quality 1S to translate furore 
needs of the user into measurable characteristics, so that 
a product can be designed and turned out to give 
Satisfaction at a price that the user will pay....The 
quality of any product or service has many scales. A 
product may get a high mark in the judgement of the 
consumer on one scale and a low mark on another. 
In direct contrast to the letter and intent of the 1984 
Competition in Contracting Act, Deming advocates limiting the 
number of suppliers to a chosen few for long term 
relationships because [Ref. 4 : p. 13]: 
We can no longer leave quality and price to the forces of 
competition = not in today's requirements for uUnwtommam, 
and reliability. Price has no meaning without a measure of 
quality being purchased. American industry and the U.S. 
Government are being rooked by the rules that award 
business to the lowest bidder. 

D.A. Garvin has the most sophisticated view of quality, 
using five different approaches for providing a framework for 
defining quality: the transcendent, product-—based, user-based, 
manufacturing-based and value-based approaches. He adds eight 
dimensions of quality to be considered by each approach: 
performance, features, reliability, Conformance, dura AEE 
serviceability, aesthetics and perceived quality [Ref. 5: p. 
101]. According to his hypothesis, the definition of quality 
is variable, depending on the definer's approach point of view 
and which critical dimensions of quality were applicable. 

From the discussion above it is obvious that there is no 
single complete, all encompassing definition of quality which 


is universally recognized and accepted in industry or 


Government, there are just too many variables and viewpoints. 


Perhaps the very best operational definition of quality in DoD 
today was offered by Dr. Robert E. Costello, the former Under 
Secretary of Defense for Acquisition, when he proposed a 
definition of quality which was accepted by both Government 
(The Defense Science Board and The Defense Manufacturing 
Board) and industry (The Manufacturing Advisory Committee) 
MRet. 6: p. 14): 


Quality ee ce conformance to COBRE cti defined 
requirements satisfying customer needs. 


Dr. Costello expanded further on this definition of quality 

Peete 62 pw 15): 
weenie ¡Composite Of “makerial wattyibutes, ineluding 
performance characteristics and features of a product or 
service to satisfy a given need. Translation: Is the 
Reeguct Good, and’ Will it~ do the obr ifor which itis 
intended? 

C. TOTAL QUALITY MANAGEMENT (TQM) 

TOM is a management StvleesCOnmstUStLing Of a set “OF 
techniques and tools for continuously improving the degree of 
quality as described in Section B, in day-to-day operations at 
every organizational level, in every area of responsibility, 
and in every product or service produced. TQM combines 
fundamental management techniques and strategies, existing 
process improvement efforts and statistical evaluation tools 
wo a dynamic, disciplined structure. -focused on continuously 


m oving all processes [Ref. 7: p..2]. TOM is based on the 


following elements [Ref. 8: p. 1]: 


le Development of a culture or environment committed 
to continuous process/product/service improvement. 


Ze: An understanding of the needs of the customer, and 
keeping those needs foremost in the organization’s 
Priorite 

oe Increased involvement by all levels in the work 
force. 

4. Establishment of a teamwork approach to solving 
problems, and the use of CroSsftunctionall 


cooperation and dialogue. 


a An understanding that most problems are a function 
of process management, and not people problems. 


Gx Making decisions based on data, not subjective 
inferences. 
das Establishment of a commitment to quality that is 


pervasive tee Olive ie the Srganization, DUE 
starting with strong support from top management. 


8. Stressing that small, constant improvements are 
Just as important as large-scale innovations. 

Within DoD, TQM implementation will focus on continuous 
process improvement of every facet of its operations, i.e. 
internal operations, weapon system requirements formulation, 
design, development, production planning, source selection, 
manufacturing, fielding and follow-on logistical support om 
9: p. ii]. The intent is for TOM to evolve beyond being Woes 
another program or buzzword, and to literally change the 
quality culture of the DoD establishment, its contractors and 
their principle “subecenerderoarse This shift of quality ocu 
translates into preventing defects through quality being 
designed-in and built-in to the produce or Service iiseead aa 


being “inspected-in" or through defect correct ormaIt a e 
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means that continuous process improvement will replace 
"acceptable levels" of defects, properly defined requirements 
will replace approval of waivers to conformance, and that the 
emphasis will change to quality, cost and schedule instead of 
ai a ost and schedule [Ref. 9: p. 111]. 

TOM is process oriented, and as such goes beyond the 
traditional DoD quality assurance system and "after the fact" 
product inspection. Some of the common tools used in the 
process of operating under TOM include: benchmarking, Cause 
and effect diagrams, -pareto charts, statistical. process 
Semerol, histograms, 1nput/output analysis, scatter diagrams, 
concurrent engineering, control charts, work flow analysis, 
team building and time management [Ref. 9: p. 50]. When fully 
implemented, TOM in the DoD establishment and military 
industrial base will encompass the following principle areas 


Peete, 9: PD. ivi: 


IF: The quality of management. Effective leadership, 
Potman ernal y Sand aat T DoD contractor» “ana 
principle SUDCOntractor levels. TOM demands 
involved, participating and high quality 


management of all processes that add up to the 
acguisition process. 


2 The quality of processes. Every functional element 
inac Ne POP and Industry -must “Continually search 
for process shortcomings and devise ways to 
gradually overcome them through a continuous 
improvement process. 


a The quality of the integrated hardware/software 
systems and services provided to our field users. 
With respect to product quality, TQM expands the 
der nieto econ the precuck COnrormmance Locus to 
one which starts with the definition of correct 


il 


requirements, then the achievement of conformance 
to these requirements through continuous process 
improvement, not exclusively through inspection. 
The bottom line will be the achievement of total 
user satisfaction. It should be clear that total 
conformance to an incorrect requirement results in 
a perfectly incorrect product or service. 


D. THE MALCOLM BALDRIDGE NATIONAL QUALITY AWARD 

One set of guidelines or criteria which can be effectively 
used to evaluate an organization’s commitment to and 
utilization of some form of TOM or quality assurancée 1n ASS 
corporate charter or philosophy, and in their day Tomo E 
management is the application and evaluation guidelines for 
the Malcolm Baldridge National Quality Award (MBNQA). These 
guidelines were jointly developed by a Government and industry 
coalition, and are widely accepted as being the most current 
and objective guidelines available today for evaluating 
corporate TQM implementation or other quality assurance 
efforts. An outline of the guidelines as of 1990 are 
contained in Appendix A. 

The MBNOA is an annual award required since the passing of 
Pai PUSO The Malcolm Baldridge National Quality 
Improvement Act of 1987. The award process is managed by the 
U.S. Department of Commerce and the National Institute of 
Standards and Technology, and is administered by the Malcolm 


Baldridge National Quality Award Consortium, formed by the 
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American Society for Quality Control and the American 
Peeauctivity and Quality Center [Ref. 10: p. 7]. 

This award recognizes any U.S. company which excels in 
quality management and quality achievement within three 
Faregories: manufacturing, service and small business. This 
award promotes and rewards [Ref. 10: p. 4]: 


lee An awareness of quality as an increasingly 
important element in competitiveness. 


Z: An understanding of the requirements for quality 
excellence. 
Sn The sharing of information on successful quality 


strategies and on the benefits derived from 

implementation of these strategies. 
These application guidelines are not only effective for use in 
gathering information for MBNQA evaluation procedures by the 
Beara Of Examiners, but they have been extremely useful to 
many companies, including McDonnell-Douglas, as a method for 
self-diagnosis of the company’s commitment to quality and 
continuous process improvement [Ref. 11]. 

As a measure of how seriously Government and industry 
takes this award, over 180,000 applications were submitted for 
the 1990 award. The award has also received high level 
dare tion from the President. of the United States to the 
Presidents and CEO’s of most firms in the U.S. today [Ref. 10: 
ee rl: 

The improvement of quality in products and the improvement 


of quality in service - these are national priorities as 
never before..... All American firms benefit by having a 


ie 


standard of excellence to match and perhaps, one day to 
surpass. There can be no higher standard of quality 
management than those provided by the winners of the 
Malcolm Baldridge National Quality Award. 
- President George Bush . 
If you intend to become the world’s best, it’s essential 
to establish what that takes, document it, take action on 
it, and then measure your progress. The Malcolm Baldridge 
1S a road map for us to achieve our vision of becoming the 
world’s preeminent space systems company. 
~ Ken Francis, President 
McDonnell-Douglas Space Systems Company 
E. THE CURRENT ATTITUDE TOWARDS QUALITY AND TQM IN THE NAVY 
Some of the key characteristics of DOD procurement 
activities in general and Navy procurement activities in 
particular are that the methods are structured in nature, 
centralized to some extent to provide consistency, and open to 
audit and review. The Navy purchasing environment is further 
impacted by many factors outside of the control of individual 
PCO’s. These factors include: socioeconomic goals, influence 
of the dynamic federal budget process with its fluctuations, 
the distinction. between price and cost, the diffiiculi eee 
establishing specifications, the distinction between "soc 
Suitable and "goldplating," the preference for fixed price 
contracts, and the strong preference for competition. Within 
this environment, the PCO must take written requirements from 
the requiring office or activity, match them with availabe 
Suppliers, and negotiate the most favorable terms for the 


purchase. Price is most often the dominant source selection 


evaluation factor in this procurement process, and may be 
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mieweq aS Stirling PCO creativity and innovation, as Dr. Harry 
Rage States [Ref. 4: p. 42]: 
It has become traditional practice in government to write 
purchase specifications in such a way that any potential 
Supplier can produce the item, and award can be based upon 
lowest price. 
With this environment in mind, what are the prevailing 
attitudes of the Navy acquisition community towards quality 
factors in the source selection process? 

CDR Charles A. Perkins, SC, USN performed an extensive 
survey of and personal interviews with Navy PCO’s, Navy 
Program Managers (PM/ s) and members of the National 
Association of Purchasing Management (NAPM) in 1988 as part of 
his dissertation research. The: “study sitselt Vconsisted .ot 
using two similar questionnaires testing seven hypotheses 
(Ref. 4: p. 41]. The first questionnaire was directed to the 
17 designated Navy Program Managers and was returned with a 
response rate of 88%. The second was sent to 517 Navy PCO's 
and 1,163 NAPM members with response rates of 45% and 37% 
respectively (statistically significant samples). Follow-up 
interviews were conducted when requested or when necessary to 
clarify expressed opinions. 

Nonparametric and exploratory data analysis statistical 
tools were used to test the hypotheses. The following are 


supported hypotheses and conclusions drawn from the analysis 


aime collected data [Ret. 4: p. 47? = 51]: 


la 


SUPPORTED HYPOTHESES: 


Competition is viewed as positively influencing 
the quality of products obtained unless the 
competition is dominated by price considerations. 


Price competition is a more significant factor in 
Navy procurement than in commercial/industrial 
pürcnacing- 


The effectiveness of the Navy procurement process 
in obtaining quality products is inadequate 
because of the overemphasis on price and the poor 
feedback of accurate and timely quality related 
information from the end-users back to the Navy 
ECO 


CONCLUSIONS: 


The comparison of the attitudes and preferences of 
the sample populations of Navy PCO’s and NAPM 
members showed a significant difference in the 
importance they place on quality factors in their 
purchase decisions, with heavier attention placed 
on it in the commercial/industrial sector. 


Satisfaction of the minimum specifications drives 
the award in price-based decisions. 


Consideration of other non-price related factors 
evokes a notion that the item is "goldplated," or 
that too much is being paid for an item. 


Interviews with Navy contracting officials 
established that a method of source selection 
based upon prestated quality measures would be 
used if a feasible, generally accepted model were 
available. 


Of CDR. Perkins” seven quality characteristics 
ranked by PCO’s, PM’s and NAPM's, performance and 
reliability were ranked number one and two in 
importance by all three groups. PCO’s ranked the 
remaining factors in order of importance as 
maintainability, schedule, durability, past 
performance and warranty respectively. PM's ranked 
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Ee 


them as durability, maintainability, past 
performance, schedule and warranty, and NAPM as 
Omaha yy schedule, maintainability, past 
performance and warranty respectively. 


FEDERAL ACQUISITION REGULATION 


FAR subpart 15.605 addresses source selection evaluation 


factors, and not only encourages quality to be an evaluation 


Malos: in all negotiated contracts, but makes it and price the 


two required evaluation factors in every source selection 


Heer. 12: p. 16,926): 


G. 


(B) The evaluation factors that apply to an acquisition and 
the relative importance of those factors are within the 
pecad discretion of agency acquisition officials.... Quality 
also shall be addressed in every source selection. In 
evaluation factors, quality may be expressed in terms of 
technical excellence, management capability, personnel 
qualifications, prior experience, personnel qualification, 
past performance and schedule compliance. 


(C) While the lowest price or lowest total cost to the 
Government is properly the deciding factor in many source 
selections, in certain acquisitions the Government may 
select the source whose proposal offers the greatest value 
to the Government in terms of performance and other factors. 


(D) In awarding a cost-reimbursement contract, the cost 
proposal should not be controlling, since advance estimates 
QE St May not. be valid indicators of final actual costs: 
Mie primary consideration should be which offeror can 
perform the contract with results most advantageous to the 
Government, as determined by evaluation of proposals 
according to the established evaluation criteria. 


THE PRIMARY FORCES INFLUENCING QUALITY PERCEPTIONS AND 
TQM UTILIZATION IN DoD SOURCE SELECTION SINCE CICA 


D7 


Since the passage of CICA in 1984, there has been a widely 
held view that defense procurement overemphasizes the 
importance of cost/price in source selections. While CICA did - 
not specifically forbid the use of quality as an evaluation 
factor in DOD procurements, the manner in which it was 
implemented by DOD, and subsequent Comptroller General 
decisions Supporting CICA both Contributed to this views 

While not openly criticizing the letter or intent of ines 
the President’s Blue Ribbon Commission on Defense Management, 
more commonly known as the Packard Commission, identified 
three problems in 1986 with CICA’s implementation in DoD [Ref. 
ato A216 

I5: the interpretation that the government must buy 
from the lowest price bidder; 

Ze the notion that CICA precludes qualificatienm 
criteria, consideration of technical expertise, or 
life cycle costs; and 

B: the resulting focus on the number of competitions 
rather than the success the competition achieves 
in terms of reduced prices or better products. 


The Packard Commission went on to make Recommendation F, 


that CICA’s full potential could not be reached Untill ieee 


l3: po TCAE 
Federal Law and DOD regulations should provide for 
Substantially increased use one commercial-style 
competition: emphasizing quality and established 


performance as well as price. Commercial procurement 
competition simultaneously pursues several related 
objectives: attracting the best qualified suppliers, 
validating product performance and quality, and securing 
the best price. Procurement officers must be allowed and 
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encouraged tOr z: give preference to suppliers that have 
demonstrated quality and reliability, and to recognize 
vale ¡quality anda” price...) based Ton a> product's 
commercial acceptance in the marketplace. Price should not 
be the sole determinant, especially for procurement of 
complex systems and services. 

This ideal was supported by RADM Robert Moore, the Navy’s 
Competition Advocate General [Ref 4: p. 50]: 

The objective is to...change the focus of Navy acquisition 
away from lowest price to a contractor’s capability to 
Produce quality products. 

On 25 February 1986, President Reagan signed Executive 
Order 12552, which required all departments of the federal 
government to improve the efficiency, effectiveness and 
quality of the products or services delivered through TOM [Ref 
7: p. 4]. The President’s productivity objectives also set a 
goal of 20% productivity improvement in appropriate functions 
by loo? . 

In September 1987, Everett Pyatt, the Assistant Secretary 
cine” Navy (Shapbuilding and Logistics), issued a policy 
memorandum on source selection procedures within the Navy 
which stated [Ref 14]: 

The goal should be to define the quality standards 
appropriate to each requirement and to communicate them 
plainly to selection officials and to all offerors. When 
the quality desired can be sufficiently defined to assure 
success and proposals can be evaluated to determine if 
they meet the specified quality, then the source selection 
Criteria should be: "award to the lowest priced (realistic 
mares tor Cost trpe) technically acceptable Obfercr. 

On 18 August 1988, then-Secretary of Defense Frank 


Carlucci announced that DOD would formally implement TOM 


miareouchout all DOD activities... He stated that TOM would 


ES 


require a complete change in our traditional approach to doing 
business [Ref 7:3 ps 2) and tae: 
I am giving top priority te the DoD Total Cua, 
Management effort as the vehicle for attaining continuous 
quality improvement in our operations, and as a major 
strategy to meet the President’s productivity objectives 
UNder er, © jets oe 
Within this overall effort, Dr. Robert Costello SE m 
former Under Secretary of Defense (Acquisition), presented 10 
major items on his 1988 defense acquisition agenda which 
included the goals of: improving product quality to reduce the 
cost of defective products and services through TOM, Mama 
achieving substantial reductions in the lifecycle cost of 
ownership of weapon systems [Ref. 6: p. 13]. Dr. Costello 
provided further amplification of his goals [Ref. 6: PIM 
We must emphasize competition based on quality as well as 
cost, schedule and performance, including lowest bid. Our 
objectives include: making quality a factor in soOUmee 
selection; giving extra consideration to companies whose 
products and services embody the new concept of continuous 
product improvement... To implement this strategy, we 
will: encourage contracting officers to look for ways to 
increase quality when preparing requests for proposals and 
negotiating contracts... Industry must provide rancia 
evidence of its commitment to quality. | 
It is not only DOD that is fast becoming an ardent 
proponent of quality as a prominent source selection 
evaluation factor, industry is also voicing its opinion Sei 
quality is a critical business issue in reforming the DOD 
acquisition system. The Navy League Executive Forum identified 
this concern Ref. o]: 
The Navy position regarding the importance of price should 


be more flexible. There are occasions when the Navy's best 
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interest would be served by deemphasizing price in the 
selection process, such as when a substantially better 
quality product is available for a relatively modest price 
premium...The Navy should develop source selection 
criteria recognizing past performance as part of "best 
value" and control the use of price as an overriding 
selection criteria. We should improve the development of 
Sormection criteria, and expand programs to collect and 
Share vendor performance data... 


H. THE ARMY’S USE OF TQM AND STATISTICAL PROCESS CONTROL 

IN SOURCE SELECTION 

One of the first extensive uses of TQM and Statistical 
Process Control (SPC) factors in a formal source selection 
procedure for a major defense acquisition was completed in 
March 1990 when the U.S. Army Communications-Electronics 
Command (ACEC) awarded two multiyear contracts to ITT and 
ean for the production of third generation night vision 
devices and tubes [Ref. 16]. The solicitation was structured 
utilizing a best value basis for award, and the Army agreed to 
a total evaluated price of $491M, which was $47M higher than 
the lowest priced combination of awards, an approximately 10% 
price premium paid through the best value decision. The Army 
considered night vision devices to be the "backbone" of its 
Meme fighting Capability, and it conducted the acquisition 
with a method which ensured that the Government would obtain 
the most advanced equipment of the highest quality with the 
best follow-on logistical support at the lowest total cost of 


Ownership [Ref. 16]. 
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There were four evaluation factors in Section M of the 
Request for Proposals (RFP): Price, Technical, Prode 
Assurance and Test (PAST) and Production and Management (P&M). 
Price was more important than any one of the other three 
factors, which were of equal weight. Combined, however, the 
other three factors were significantly more important than 
price. Additionally, the RFP explained that past performance 
would be separately evaluated to assist in determining overall 
performance risk [Ref. 16]. 

In conjunction with the basic proposal evaluation, SPC was 
evaluated as one of the five subfactors within the PAST 
factor, and TOM was evaluated as one of four subfactors within 
the P&M factor. In addition, both TOM and SPC were also 
independently verified as part of the separate past 
performance/performance risk assessment (PP/PRA). ACEC was 
careful not to mention specific TQM or SPC requirements or 
methods in the solicitation, but mentioned that the Source 
Selection Evaluation Board members would analyze how offerors 
would employ their TOM and SPC techniques to facilitate 
quality performance on this particular contract. Addition TERE 
the SSEB would assess the offeror's management commitment to 
timely production and delivery of only the highest quality 
equipment [Ref. 16]. | 

ACEC conducted several industrywide briefings, culminating 
in formal presolicitation and preproposal conferences. Great 


pains were taken to ensure that the competitors understood how 
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the evaluation and selection process would be conducted. The 
competitors were given ample opportunity to explore and 
resolve any and all issues [Ref. 16]. 

ACEC’s philosophy was that a contractor’s past performance 
Recordad 15 a strong indicator of its ability to successfully 
perform in the present and future. Therefore, the offerors 
were informed that a detailed review of particular areas of 
past performance would be evaluated to assess performance 
risk. The evaluation of past performance was conducted by a 
separate evaluation team within the SSEB. The evaluation 
encompassed information provided by the offerors in their 
proposals as well as extrinsic data, and focused on the extent 
to which the offerors had previously utilized SPC and TQM 
techniques and the results attributable to such efforts [Ref. 
Tol. 

The results of the basic proposal evaluation for each 
factor (other than price) were then correlated by the SSEB 
chairman with the PRA for each factor to provide the overall 
factor ratings. Ratings for the basic evaluation and the 
overall rating were: outstanding, acceptable and marginally 
acceptable, and the PP/PRA ratings were: low, moderate and 
high risk. No numerical scores or weightings were used, and 
the blending of the basic evaluation rating and the PP/PRA 
meeting ¿into an overall rating for each non=price factor was 


somewhat subjective. For example, an "“acceptable" basic 
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evaluation rating for PA&T may be raised by a "low"  PP/PRA 
rating to produce an "outstanding" overall rating [Ref. E TE 

The award was immediately protested by Litton to the. 
General Accounting Office (GAO) based on the issue of 
ACEC’s accuracy of the PP/PRA ratings for TOM and SEC. Fic emi 
alleged that ACEC placed undue emphasis on TQM and SPC ina 
manner inconsistent with Section M of the RFP, and that TQM 
and SPC were undefinitized "paper philosophies" for which 
there was no formal Army guidance. In ruling in favor of the 
Army, GAO addressed and rejected each of the allegations, a 
decision later affirmed by the Federal District Court. 
Consequently, the validity of the evaluation of TOM and SPC in 
the DoD source selection process was carefully reviewed and 
confirmed, provided the source selection process is conducted 
consistent with the process described in the solicitation 
Re. 16]. 

GAO! s decision contained references to ITT's mature, well 
run TOM program that provided a process to ensure continuous 
quality improvements and delivery of only the highest quality 
products to the Government. The decision also stated that 
ACEC’s award determination was consistent with the evaluation 
criteria, and it was in the best interest of not only the 
Government, but the soldiers and aviators whose lives depend 
on the quality of the goggles. A definitive determination of 
whether this source selection process was successful or not 


cannot be made at this time. No statistical or quality data 
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AS com production on this contract or from user feedback 
is currently available to verify whether the 10% price premium 


paid was cost effective [Ref. 17]. 


I. A DoD INITIATIVE TO INTEGRATE TQM INTO SOURCE SELECTION 

In 1989, the former Under Secretary of Defense for 
Peamrsition, the Honorable John A. Betti, tasked the Army, 
Ney fir Force and the Office of the Secretary of Defense 
(OSD) to form a Process Action Team (PAT) to examine 
Sie r native means for integrating TOM into the source 
Selection process for all of DoD [Ref. 7]. The trend now was 
to not only support the implementation of TOM throughout the 
defense establishment, but to encourage and motivate the 
defense industrial base to adopt some form of TOM into its day 
to day operations. 

In May 1990, this PAT completed development of draft 
guidance which provides program managers, contracting officers 
and source selection personnel with practical guidelines for 
making TOM a consideration in source selection. The Draft 
Guide to Integrating Total Quality Management into Source 
Selection was developed to foster <Consicuency “among 
Services/Agencies in integrating TQM into source selection, 
but was designed to afford the PCO the flexibility to tailor 
and adopt the approach to meet specific program needs [Ref. 8: 
mel). The PAT’s concept is to accomplish the goal of taking 


MICO ACC ount OT eror" sS continuous process improvement 


Ze 


activities by treating continuous process improvement as part 
of the SSEB’s PRA. The PRA provides a confidence measure to 
the Source Selection Activity (SSA) of the offeror’s potential - 
to perform on the proposed contract. 

The Draft Guidance referred to the use of a solicitation 
provision based on a modified version of the Malcolm Baldridge 
National Quality Award (MBNOQA) guidelines to obtain the 
information for the assessment. Currently there is no modified 
MBNQA guidelines for use as solicitation provisions or as 
source selection evaluation criteria due to an indefinite halt 
in the finalization of the Draft guidance into a formal 


instruction Ref. ae 


J. SUMMARY 

After discussion of what quality and TQM are generally 
accepted as being defined as, evidence was presented that the 
DoD acquisition community has the desire, incentive and 
mandate to integrate TOM or other quality factors into source 
selection. Evidence was also provided that the MBNQA 
guidelines are a viable information collection and assessment 
tocl to gauge corporate adoption of some form of TOM is 
continuous process improvement in its operations and products 
or services. 

The next chapter will describe the DOD/ JOA 
Service/Agency PAT Draft Guide in more detail, and will 


present the researcher's source selection procedure model. 
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The flexible model will be a synthesis of the ACEC’s method of 
procurement for the third generation night vision devices with 
the evaluation guidelines or criteria suggested by the PAT in 


their Draft Guide. 
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III. A SOURCE SELECTION PROCEDURES MODEL 


A. INTRODUCTION 

Chapter II presented definitions of quality and TOM, a 
description of the Malcolm Baldridge National Quality Award, 
evidence of willingness among Government PCO's to use quality 
factors in their source selection processes, the successful 
use of TOM and SPC evaluation factors by the Army in a major 
acquisition, and a general overview of a current DoD/Joint 
Service/Agency initiative to develop and promulgate a guide 
for integrating TOM into source selection. This chapter will 
describe the DoD initiative in more detail, and will present 
this researcher's source selection procedures model. 
B. OVERVIEW OF THE DRAFT GUIDE TO INTEGRATING TQM 

INTO DoD SOURCE SELECTION PROCEDURES 

The DoD/Joint Service/Agency Process Action Team draft 
working paper of a Guide to Integrating Total Quality 
Management (Draft Guide) was given a limited distribution in 
DoD for the collection of constructive feedback from selected 
services and agencies on May 8, 1990. The Draft Guide was 
Subsequently revised and distributed for review and comment 
again on November 5, 1990. Appendix B consists of sections mammi 
through VII of the November 5, 1990 draft working paper. 


Sections I and II are the Introduction and the Concept 
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portions respectively, and they are summarized in 
remainder of this section. 

The Draft Guide was developed to foster consistency among 
the DoD services and agencies by recommending an approach for 
the inclusion of TQM factors into the source selection process 
De n S and PM's [Ref. 19. : p:. 1]. lt 1s primarily applicable 
to major defense system acquisitions covered by DoD Directive 
5000.1 (Major and Non-Major Defense Acquisition Programs), but 
maybe used in any other DoD acquisitions at the discretion of 
Pee Service or agency [Ref.19: p.1]. The recommended 
procedures also allow PCO’s the flexibility to tailor or adapt 
the concepts to meet the specific needs of particular 
Seguls it ons . 

ie principal rationale for the development and 
promulgation of the Draft Guide is that offerors are most 
INS” to provide high quality, on-time, within=cost products 
and services if they have viable continuous improvement 
efforts backed up with demonstrated results [Ref. 19]. The 
goal of the Guide is to take into account the maturity and 
effectiveness of the offeror’s continuous process improvement 
(CPI) programs as part of the source selection process [Ref. 
19: p. 2]. Primary emphasis is placed on tangible evidence and 
verifiable proof that these CPI vehicles are in place and 
functioning effectively in time for the execution of the 


Mroposed contract, not just Existing on paper or brochures. 
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The key strategy for achieving the goals and intentions of 
the Draft Guide is treating CPI and offeror past performance 
as part of the SSEB’s estimate of the risk involved in making . 
a contract award to a given offeror. An REP provision based 
primarily on MBNQA guidelines being modified through a 
reduction in scope and detail is used to obtain offeror 
information for this assessment. Also included in the risk 
assessment is an evaluation of the offeror’s past and present 
performance record [Ref. 19 pe Zils Together, these 
evaluations by SSEB teams comprise a PRA, and the 
recommendations of the SSEB are forwarded to the SSAC for 
further evaluation. The source selection organization to 
which the Draft Guide and the recommendations of this thesis 
are primarily applicable to is the standard organization for 


most major weapon system acquisitions. [Ref. 20]. 


Source 
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C. SUGGESTIONS FOR FINALIZATION OF THE DRAFT GUIDE 

mele Opinion Of this researcher, the Draft Guide could 
have offered more in-depth discussion on the following key 
areas for PM/PCO guidance or consideration: 

T Actual examples of solicitation guidelines derived 
from modified MBNOA guidelines; 

om specific recommendations for scoring or rating 
techniques or other quantitative evaluation 
Standards £Or -the SSEB to utilize in evaluating 
the offers and extrinsic evidence, other than the 
use of forwarding “narrative comments" to the SSAC 
on offeror strengths and weaknesses; 

2: recommendations LO a delineation Gar 
responsibilities for evaluation of the basic 
proposal cana. Lor. the ~conduct —~ of, a. PRA, aña 
references to specific sources of extrinsic data 
for use in validating past performance in the PRA, 
other than: "discussions I withetecnniical cana 
PLCgdram Managers... and. <.from actual, perrormance 
data of products, where available." 

In addition to the three areas found by this researcher to 
be in need of clarification or amplification, there is one 
procedure in the Draft Guide which this researcher recommends 
to be modified based on the results of the Army procurement 
described in Chapter II section G. This was one of the first 
cases where a service or agency has extensively utilized TOM 
pilosa? C AS Critical subfactors in a formal source selection 
Procedure. ACEC*s inclusion of TOM and SPC subfactors in the 
Paste proposal evaluation as well as TOM, SPC and past 


performance areas in the PRA were supported by G.A.O. and 


Mederal District Court decisións since ACEC was Consistent in 
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performing the proposal evaluation and PRA in accordance with 
the terms in the RFP. Although the DoD/Joint Service/Agency 
PAT Draft Guide and the ACEC’s third generation night virour 
device source selection procedure are similar in some 
respects, they are fundamentally different in other areas 
which have important implications for the extensive DoD-wide 
use of the Draft Guide in its current form. The most come 
aspect in which the ACEC's procedures differ from the PAT 
Draft Günde consists Co: 
ay ACEC's use of TOM, SPC and past performance as 
subfactors set forth in Section "M" of the Request 
for Proposals (REP) tolbe evaluated an Mra en 
along with the other subfactors within the 
Technical, PA&T and P&M factors. In addition, TOM, 
SPC and past performance were also evaluated in 
conjunction with the PRA. The adjectival rating 
from the basic proposal evaluation was tempered 
with an adjectival PRA rating by the SSEB Chairman 
to produce an overall rating for each factor ven 
each proposal. 
The remainder of this chapter will Consi CEE 
recommendations for a revision of the Draft Guide based on the 
four areas listed above, including a proposal evaluation 


guideline model for use by the SSEB to evaluate and rate 


offeror response to Soc Cion e REE 


D. RECOMMENDATIONS 
1. Appendix C consists of a proposal evaluation model for 
the SSEB to utilize when evaluating, rating and scoring the 


sections of the offered proposals which address the CPI 


SZ 


A eon Of REP Section "L" in Appendix B. It is sufficiently 
flexible enough to allow the SSEB to use either a numerical 
scoring system, an adjectival rating system, a narrative 
system utilizing summaries of strengths and weaknesses, or 
some combination of scores, adjectival ratings or narrative 
descriptions. The subordinate subfactors themselves are 
derived from the MBNOQA guidelines, which have been: 
a. modified into evaluation criteria with a DoD 
orientation i.e. DoD terms, special interest 


areas, and actual examples of items to give credit 
for while evaluating. 


e reorganized into a more logical sequence for 
evaluation; 
on modified to reduce overlapping and "double- 


weighting" in the evaluation process through a 
consolidation of related themes; 


d: streamlined through deletion of specific subareas 
within certain guidelines because they were either 
inapplicable to - DoD procurement, were ECO 
Subjective, or were potentially too difficult to 
validate with extrinsic past performance data. 

Source selection personnel must be aware that they can, 
and in many cases, should modify, expand, reduce or otherwise 
tailor the language contained in the subordinate subfactors of 
Appendix C. The "factor" heading has not been filled out in 
order to allow flexibility in assigning these evaluation 
Subfactors to factors such as Technical, Management, Product 
Assurance and Test, Production and Management, Integrated 


Mogisctic Support or whatever the contracting activity” s 


SclilciLacion Organization. dictates. 
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2. In order to make the Appendix C proposal evalua 
model a useful tool for the SSEB, Sections III and IV o Eme 
Draft Guide should be modified. The Acquisition Plan (AP) and - 
Source Selection Plan (SSP) in Section III of Append EE 
should be changed from a reliance on the conduct of a PRA 
only, to an adoption of the ACEC strategy of making TOME T 
and past performance rated factors in the basic proposal 
evaluation as well as rated factors in the PRA. 

Also in Appendix B, the sample language in the Section IV 
Executive Summary and in Sections "L" and "M" of the RFP 
should be modified to inform the offerors that the Government 
intends to consider TOM, SPC and past performance as not only 
evaluation factors in the basic proposal evaluation, but as 
factors in a PRA performed separately and independently of the 
basic proposal evaluation. The recommended procedures for 
conducting the PRA will be addressed in Chapter IV of this 
thesis. 

In order to provide an added degree of assurance that the 
offerors understand exactly what is required of them to be 
fully responsive to the solicitation requirements, the opening 
remarks of subsection 2.70 - Specifie Information and Data 
Section IV, should be expanded to include the follan 
instructions, which are modifications of the gquidelinest@imes 
MBNQA applicants when filling out their applications [Ref. 


MO 
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Read all of the solicitation categories and 
specific items within the categories before 
developing responses to any of them. 


Interpret the meaning of each specific item within 
the context of the main purpose of the category. 


neck celatrece items to determine the distinctions 
between the types and extent of information 
required. 


Respond to items with concise, factual statements. 
Support statements with quantitative information 
whenever appropriate. Use of graphs, charts, and 
tables, properly labeled and compactly presented 
is strongly encouraged. Lengthy narratives not 
directly responsive to the purpose and substance 
of an item are discouraged. Also, avoid the 
reiteration of the words and phrases of the 
Solicitatton, tems by using “your “own: original 
words and phrases. 


Avoid the use of anecdotal information or 
information lacking overall context. An example is 
at times appropriate, but offerors should make 
clear that the example illustrates the larger 
point being made, and is not itself the response 
co the ten: 


Report only what is requested in each item, and 
include only the types of information requested. 
Make responses self-contained, and not dependent 
Moen Jn ro rmac lon. given. in responses. VEG) other 
items. However, 1f other items contain information 
eae willl nel ate “elarity «Or, “strengenens. «a 
response, and at the same time, avoid significant 
duplication OT terms Lon; provide Cross 
references. 


Trend data are requested to permit applicants to 
demonstrate progress and to show that improvements 
are sustained. No minimum period of time is 
specified for trend data. Time periods for trend 
data may span up to five years or more for some 
product... Service. or Operational -characteristics, 
but may be much shorter in areas where improvement 
efforts have been established more recently. 
Trend data should be presented in graphical, 
tebularrer ciher- compact form. include appropriate 
benchmarks or other references that help to ensure 
the proper ¿interpretation oí cata. 


BS 


ig Identify responses to categories and items within 
categories with the letters GE numbers 
corresponding to each category and items within 
each category. 

The actual solicitation categories and items within each 
category listed in the RFP subsection 2.0 — Specials 
Information and Data, should be the same as the wording of the 
evaluation factors and subfactors in Appendix C, instead of 
the wording of subsection 2.0 of the Draft Guide. This TEESE 
ensure that proposals are evaluated by the same criteria that 
are Set forth Gamwrene Solicitare nk 

3. The Draft Guide did not make specific recommendations 
for scoring or rating techniques, or other quantitative 
evaluation standards for the SSEB to use while performing the 
basic proposal evaluation or the PRA. This was the DoD/Joint 
Service/Agency PAT’s intent because it would not be desirable 
to standardize the rating/scoring techniques throughout DoD. 
Maximum flexibility must be allowed in this area if it is to 
be of value to DoD services and agencies. 

Based on the ACEC procurement, this researcher recommends 
a specific proposal and PRA evaluation method which utilizes 
a combination of the numerical scoring, narrative description 
and adjective grading techniques. Appendix D contains examples 
of proposal and PRA evaluation worksheets utilizing the 
Continuous Process Improvement of Processes, Products and 
Service component of the Quality Assurance of Products “ama 


Services subfactor. The XYZ Corporation is actually another 
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Mame enor a “real firm which submitted a proposal to ACEC in 
besponse to their Solicitation for the third generation night 
vision devices. The proposal evaluated in this example was the 
elsa proposal submitted by this firm. The maximum points 
column available for each subfactor item is filled in based on 
the relative weighting assigned by the SSEB. The assessment 
percentage is based on the overall scoring justification. The 
percentage is based on the evaluator’s best assessment of the 
offeror’s fulfillment of the requirement on the following 


scale developed by this researcher: 


Percentage: 


Superior 90 1100003 
Acceptable IU Eos 
Marginal SUF = soos 
Unsatisfactory OPS Ol 


The percentage selected should be justified in the "overall 
rationale for assessment %" section of the worksheet, based on 
a relative weighing of the strengths versus weaknesses. 

Once the scores from the evaluation of each subfactor are 
completed for both the basic proposal evaluation and the PRA, 
the raw scores should then be summarized on a consolidated 
tally sheet, added together, then converted into an adjectival 
grade and risk level corresponding to whatever groupings of 
factors/subfactors the SSEB determines as logical, feasible or 
Practical. 

An example of this is provided in Appendix E. Here, the 


= E has -consoligated the subfactor/factor scores into one 


ou 


numerical score each for the Leadership and Personnel 
Management and the Continuous Process Improvement evaluation 
factors, although they could have grouped them in any manner - 
which they deemed most useful. The raw scores are then 
converted into an adjectival rating and risk level for the 
basic proposal evaluation and PRA respectively. The final step 
is the "tempering" of the basic proposal evaluation” Tata 
with the corresponding PRA risk level into an overall rating 
for each factor. 

Once the overall rating for each factor is determined, it 
1S submitted from the SSEB to the SSAC along with any required 
supporting documentation for inclusion in a final award 
decision process. The rationale for the conversion of raw 
scores into adjectival ratings is that the numerical scores 
may be more difficult to justify to the SSAC, or to delen SENi 
the possible event of a GAO protest from an offeror or an 
interested third party. 

4. The specific details of the conduct of the PRA by the 
SSEB, including recommended sources of extrinsic past 
performance data and recommended procedures for determining 


performance risk levels are contained in Chapter IV. 


E. SUMMARY 
This chapter described the DoD/Joint Service/Agency PAT 
Draft Guide in more detail, offered suggestions for 


modifications to be made to the Draft Guide before its 


oe 


possible finalization, and presented a source selection 
procedures model consisting of a basic proposal evaluation 
integrated with a PRA. The next chapter will examine the 
Performance Risk Assessment process more thoroughly, and will 
present some of the current sources of contractor historical 


quality history available in DoD for the conduct of PRA’s. 
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IV. PERFORMANCE RISK ASSESSMENT 


A. INTRODUCTION 

Chapter III and Appendices C,D, and E presented a source 
selection model consisting of a procedure for gathering 
offeror information through RFP language and solicitation 
provisions on an offeror’s current TOM system or ICS m 
version of TOM, and the proposed methods the offeror intends 
to utilize in applying their continuous process improvement 
program to the proposed contract. Their proposal would then be 
evaluated and rated, and the resulting adjectival rating for 
a TOM related factor would then be either raised, lowered or 
remain unchanged by a corresponding level of performance risk 
into an overall rating for that factor. The basic evaluation 
of the offeror’s proposal determines proposal risk, while the 
evaluation of the actual past performance of the offeror on 
related prior contracts or similar products/services 
determines performance risk. 

This chapter will: 1. examine performance risk, 2. address 
the legal aspects of utilizing past performance data in 
determining performance risk, 3. describe a recommended five 
step procedure in performing a ES Risk Assessment, 


and 4. will list and describe some of the various primary and 
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secondary sources of past performance data available to DoD 


Peecurement Officials today. 


B. PROPOSAL VERSUS PERFORMANCE RISK 
When conducting the source selection process for major 

weapon system procurements, Research and Development efforts, 
large services contracts or selected other non-major purchases 
which involve complexities or advanced technology or 
technological risk, the Government must assess the relative 
risks associated with each offeror and proposal. These risks 
may be classified as being either a proposal risk or a 
pertormance risk [Ref. 21: p. B-2 ]: 

Proposal risks are those associated with an offeror's 

proposed approach in meeting Government requirements. 

Proposal risk is assessed by the SSEB and is integrated 

into the rating of each specific evaluation factor and 

SUDtactor. 

Performance risks are those associated with an offeror’s 

ability to perform the solicitation’s requirements as 

dicated by that offeror’s record of past and current 

performance. Performance risk may be assessed by an SSEB 

team independent of and distinct from the team assessing 

the proposal risk, or it may be assessed by a Performance 

Risk Analysis Group (PRAG) Separate from the SSEB. 
on THE IMPORTANCE OF ASSESSING PAST PERFORMANCE IN A PRA 

It has often been said that, whether it be in reference to 

the history of the world or in DoD acquisition, that the past 
is prologue. Edward J. Korte, the U.S. Army Materiel Command 
Counsel said it best [Ref. 22]: 


It has been said that history repeats itself, and that the 
person who does not study the past and learn its lessons 
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will likely be given another, often more painful 
opportunity to learn. This observation is also true in the 
world of Government contracts. For example, a failure to 
adequately address a contractor’s poor performance 
invariably haunts our future procurements. When we fail to 
deal effectively with a chronic, poor performer, we are 
often later confronted with the same problem, frequently 
compounded, and much more difficult to resolve. 


DoD Directive 4105.62, "Selection of Contractual Soca 
for Major Defense Systems" states that [Ref. 21: p. 2]" the 
offeror’s recent and relevant past performance (measures by 
such indicators as quality, timeliness, cost, schedule, 
operational effectiveness and suitability) may be considered 
in assessing the probability Of successful accomplishment of 
the proposed effort in a timely, cost-effective manner." 
Further justification for the assessment of past performance 
in source selection decisions is provided in DoDD 4105.62 
[Ref Ze ee 

Proposal evaluators must consider the technical, schedule, 
operational readiness and support, and financial risks 
inherent in a proposal. One means of assessing that risk 
is to review an offeror’s recent actual performance in 
relevant areas. Past performance as an element of risk 
analysis, may be used as one predictor of the probability 
of satisfactory performance on the proposed program being 
evaluated. Evidence of past performance may be obtained 
from numerous sources, such as the offeror’s preaward 
Surveys, onsite Government personnel at a contractor’s 
facility, field data collection systems, and other 
procuring activities that are or were customers of the 
offeror whose proposal is being evaluated. 

As mentioned in Chapter II Section F (B), the FAR also 


requires that quality be addressed in every source selection, 


and specifically that in evaluation factors [Ref. 12: p. 
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erizo]. "quality may be expressed in terms of s.: pele 


experience.... past performance and schedule compliance." 


De BASIC ELEMENTS AND LEGAL ASPECTS OF THE PRA 

Past performance evaluations in federal acquisitions are 
Paco tunately neither commonplace nor consistent [Ref. 22: 
p.1]. Some procurement activities hesitate to consider past 
performance for fear of injecting bias into the evaluation 
process, others hesitate due to a general uncertainty in the 
interpretation of the law. Many other procurement activities 
see the obvious value in past performance evaluations, but are 
unsure of the PRA procedure. Those activities which do 
evaluate past performance rely almost exclusively upon 
contractor supplied data rather than on independent data 
otherwise available to the Government [Ref. 22: p. 1]. 

Some of the hesitation and confusion associated with past 
performance evaluations can be alleviated by reviewing the 
basic elements and issues involved, and by understanding the 
legal aspects and precedents. 

1. Past Performance vs Preaward Survey 

Past performance evaluations are not the same as 
preaward surveys, as each serves a different purpose. Preaward 
surveys are conducted to determine whether a contractor 1s 
responsible, or whether the contractor is capable of doing the 
job. Past performance evaluations, such as PRA’S, are 


conducted as a part of the source selection process in 
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negotiated procurements to determine if a contractor 
acceptable. The determination answers the question, will he do 
the job successfully? [Ref. 21: p. 1]. Preaward surveys are * 
conducted by the contract administration office, and based on 
on-Site inspection or information on hand, an estimate of the 
offeror’s capability to perform a particular contr 
determined. The past performance evaluation is a very specific 
endeavor performed by the procurement contracting office to 
identify the degree of risk associated with each offeror, and 
rely’s solely on each offeror’s track record on previous 
contractual efforts [Ref. 21: p. 1]. If properly cond i 
the past performance evaluation and the preaward Survey will 
Supplement each other and provide a more complete picture of 
an offeror’s capability than either one could by itse ip 
2. Key Features of a PRA 
There are basically three key features to consider 
when performing a past performance evaluation or PRA [Ref. 21: 
DER IE 
iE Extrinsic information. A thorough evaluation of 
past performance will include evaluation not only 
of information provided ian proposals, but Valse 
information obtained from other sources. When this 
is done, the solicitation should clearly advise 
offerors, among other things, that the Government 


intends to consider information outside of the 
proposal (Extrinsic infteumaeram).. 


is. Separate evaluators. To guard against injecting 
undue bias into the evaluation process, 


reasonable/sound practice suggests separation of 
those evaluators who evaluate performance risk 
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from those evaluators who evaluate the basic 
proposal factors of cost,technical, management, 
etc. An important function of this independent 
assessment group is to insure that only relevant, 
recent and accurate past performance information 
is considered in the evaluation. 


Boi Centralized starting point. Evaluators must have 
available to them a central system that 


expeditiously provides them sources of information 

other than these cited in the proposals. These 

systems of data collection, storage and retrieval 

are COvered im Section F of this chapter. 

3. GAO 

GAO will give great deference to the procurement 
agency’s perception of a contractor's past performance 
provided that the Government follows the ground rules set 
forth in the solicitation, and reasonably considers the 
Memeractor’s data [Ref. 22: p. 2]. The Government clearly has 
the right to consider information outside of an offeror’s 
proposal to evaluate past performance [Ref. 22: p. 2]. Past 
Performance is a responsibility-type criteria, and it may be 
used as an evaluation criteria in source selections only to 
make a graded assessment of the relative merits of an 
individual proposal, or to determine the likelihood that a 
Paseroular offeror will successfully perform a contract based 
Pieeie + contractor's record under other contracts (i.e. a PRA). 
If past performance is used to determine the acceptability or 
Mm@aceceDptability of a proposal on a "pass/fail" basis, an 
unacceptability determination will likely be treated by GAO as 


tantamount to a non-responsibility determination [Ref. 22: p. 
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e AS is the case aigi making non-responsi bilge, 
determinations, the ground rules for considering performance 
data in the source selection process are that the extrinsic . 
data must be current, accurate and relevant to the contraen 
being considered for award [Ref. 22: p.3 ]. 

The Government is not required to allow offerors to 
review and rebut the sources or references which the 
Government utilizes when evaluating past performance. However, 
the Government should take every reasonable measure to insure 
the accuracy of the data relied upon during the PRA. To the 
extent that the extrinsic data gives rise to "deficiencies," 
the Government has the normal duty to disclose such 
deficiencies along with any others found during the evaluation 
process to those offerors within the competitive range for 
negotiations [Ref. 22: p. 2]. Also, the evaluation of a 
contractor’s past performance does not relieve the contractor 
of its burden of proving the technical acceptability of its 
proposal. Stated differently, the Government is under no 
obligation to seek out extrinsic data to cure deficiencies 


contained in a proposa mike oumo 


E. CONDUCT OF A PERFORMANCE RISK ASSESSMENT (PRA) 

A PRA is a process designed to determine a confidence 
measure of an offeror's potential to perform the requirements 
of a solicitation. Generally, it includes a review of each 


offeror's past performance and an assessment of the current 
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risk each offeror poses based on how they have performed in 
the past. This analysis may be conducted by a team within the 
SSEB, but separate from the proposal cost/technical evaluation - 
Eeameas recommended by the DoD/Joint Service/Agency PAT, or it 
may be conducted by a group totally separate from the SSEB, 
called the Performance Risk Assessment Group (PRAG) as 
recommended by the ACEC. In either case, the results of the 
PRA are forwarded to either the SSAC, the SSA or the PCO, 
depending on the organization of the procuring agency. The 
mone, SSA or PCO then utilize this data in concert with the 
other evaluation results to determine overall factor ratings 
and competitive ranges, to determine weakness, deficiencies 
and questions for negotiations, and, finally, to select the 
awardee [Ref. 21: P. E-3 ]. 

The SSEB PRA team or the PRAG should consist of personnel 
with technical and/or contractual expertise, as well as skills 
Ha he tollowing areas [Ref. 21: p. E-4]: 

ie Ability to conduct meaningful interviews in an 
unregulated, open-ended environment. 

ae Ability to assimilate voluminous amounts of data, 
arrive at -appropriate conclusions, cand Communicate 
reserconelasvens erxtectively both orally ana. in 
wes ngs 

oe Ability to evaluate an offeror’s TQM system or its 
equivalent based on past performance data, and 
identify and assess the principles, practices, 


tools and technigues of continuous process 
improvement. 
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It is also highly recommended that each member of the PRA team 
receive formal training in area 3 above through an in-depth 
TQM training course, such as the one offered at the Defense - 
Systems Management College (DSMC) [Ref. 19: p. 9]. 

The necessary data and sources of data to perform the PRA 
1S covered in more detail by Sections E and F of this chapter. 
Once the PRA team is assembled and trained, the information 
gathered on all the offerors within the competitive range is 
assessed through a five step process to develop performance 
risk ratings" 

STEP 1. The very first step the PRA team should take is to 
establish a methodology for assessing the degree of risk and 
to identify the advantages and disadvantages of the 
performance of this contract by a given offeror based upon his 
past performance. 

STEP 2. Next, each member should become very familiar with 
the statement of work and specifications in the RFP. 

STEP 3. In the next step, they should sift through the 
collected information and data to determine the relevancy of 
present and past contract efforts to the solicitation 
requirements. It is important for the PRA team to consider, at 
this stage, the fact that information pertaining to the 
proposing prime contractor division may not be broad enough in 
scope 1f other divisions, affiliates, subsidiaries, critical 


subcontractors or teaming contractors also perform a critical 
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EN nént Of the. contract, or significantly influence the 
proposed effort [Ref. 21: p. 6]. 

PRA’s should consider both the number and the severity of 
problems, the effectiveness of the corrective actions taken, 
and the overall work record of the offeror and the other key 
participants [Ref. 21: p. 6]. The assessment of performance 
Si 1s not intended to be a simple arithmetic function of an 
Bereror’s performance on a list of prior contracts. The 
information deemed most relevant and significant by the PRAG 
should receive the greatest consideration, or the most weight 
Meet. 21: p. 6]. The degree to which the offeror has 
demonstrated that past and present problems have been resolved 
or that planned performance has been achieved will be an 
maaicator Of the ability of the contractor to successfully 
achieve planned performance on the contract [Ref. 19: p. 8). 
The following definitions of performance risk are the most 
Beommonly used and most highly recommended [Ref. 21: p. 7]: 

TE Hron risk- Significant doubt exists, based on the 
OE teros e pertormance record, Lnat the Gtreror can 
PeLrEo rm Ene proposed- effort: 

2 Moderate risk- Some doubt exists, based on the 
OL eror Ss, perrormance record, Ena ne Otteror can 
perform the proposed effort. 

on low Erse= battle doubt. [exists, based..on the 
offeror’s performance record, that the offeror can 


Per tom —ie. "proposed Errore. 


4. Unknown — (neutral) risk- No performance record 
identifiable. 
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A basic rule of thumb for the assignment of the above risk 
levels is: if an offeror has performed consistently well in 
the past, then that offeror would be considered a low risk - 
with regard to the instant procurement. If an offeror’s 
previous performance has been inconsistent, then patterns 
leading to that spotty performance should be analyzed and a 
correspondingly higher risk assessment made. An offeror who 
has demonstrated consistently poor performance would be 
considered- -a high risk (Renee. PTT RIJ: 

STEP 5. In the event that performance data gives rise to 
a "deficiency," the PRA group should provide the contrac cama 
officer with a summary of derogatory performance information 
collected for each offeror for discussions l|Ref. 22: p MN 
As another means for insuring the accuracy of the data 
collected, the contracting officer,in conducting discussmemy 
Should provide the offeror’s the opportunity to comment on 
these deficiencies. 

STEP 6. The final step should be the complete 
documentation of the PRA effort, and the forwarding of these 
documents to the app propriar S -, SSA or PCO] Seine 
documentation should address [Ref. 21: p. 7]: 1. the sources 
of performance data, 2. the relevancy and significance of the 
data, 3. the currency of the data, 4. the Performance Risk 
Assessment of each offeror, and 5. the supporting rationale 
for each performance assessment. It is important to note that 


all performance data gathered by or included in documentation 
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used by the PRA team, is considered as source selection 
MeorMation, and should be protected against unauthorized 


peescelosure (Ref, 21: p. 7]. 


E, PRIMARY SOURCES OF PERFORMANCE DATA FOR THE PRA 
There are numerous sources of contractor present or past 
performance information and data for the PRA teams today. 
Some sources are centralized data bases for a service, agency, 
of DoD wide use, and are known as primary sources. Some are 
weeemerali zed into various local contracting activity files or 
DCAS audits/on-site visits, and some are telephone interviews 
Or written inquiries from knowledgeable points of contact, 
known as secondary sources. Before a more in-depth review is 
made of these sources of information and/or data, a 
Clarification of the difference between a past performance 
application system and a past performance data base must be 
made [Ref. 23]: 
Contractor Past Performance Application System (AS)-- any 
system that uses contractor past performance data in Support 


Om vard decisions for a. specific source selection ‘or 
procurement. 


Contractor Past Performance Data Base System (DBAS)-- any 


Syoeem for the primary purpose of the collection of 
GemeraclLor past performance data for general use. 


The following is the most current inventory of DoD 
contractor past performance data collection initiatives by 


service or Agency, with a coding as to whether they are a 
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contractor past performance data base system or a mixture of 
both a data base system and an application system [Ref. 23]: 


NAVY 


eS Navy Product Deficiency Reporting and Evaluation 
system: (DBAS) An automated Navy central data base 


system for recording information on the histor 
quality performance of Navy contractors. This 
system is supported by a group of internal (i.e. 
Contractor Evaluation System, Quality Delici- noi 
Report System, Report of Discrepancy System) and 
external (i.e. DLA Alert List, DLA Method C DE 
Program) feeder data bases, and is managed by 
NMQAO in Portsmouth, New Hampshire. 


ARMY 


ae Army Contractor Toformat ion System (CISSI E 
(DBAS/AS) Contains both a data system (Contractor 


Information Reports) and a corresponding 
application methodology. A CIR contains 
administrative data, a "network" of points jes 
contact, and alerts to quality, delivery, and cost 
performance problems. GCIR's are used during 
source selection to supplement contractor-supplied 
information and data from other government sources 
to support a performance risk analysis. 


oe Army Contractors Reguiring Special Attention 
(CRSA): (DBAS) A data base implemented by the Army 


Materiel Command which identifies and gives 
special attention to a contractor experiencing 
problems in delivery delinquencies, quality and/or 
cost. Contractors are notified when placed on the 
program and are tasked with developing "get-well" 
plans. 


4. Arm Commodi & Command Standard System (CCSS): 
(DBAS) This is the Army’s wholesale logistics data 
base management system which operates at the Army 
Materiel Command’s six commodity commands. It 
links financial management, material management, 
and contracting iIntGrmat ion sinteo  Untived™ aaa 
bases. Available data relevant to performance 
include contractor delivery and award history. 


SZ 


Armo ontra c tOr Pertormince Certiblcar ion Proaram 
(CP/2) 2 “(DBAS) An automated data base that tracks 


Contractors who are voluntarily enrolled in (CP/2Z) 
and/or certified for good quality and production 
performance. Certified status is granted after the 
contractor successfully achieves milestones which 
show improvement in process control, manufacturing 
Grecipline Vand “produc. quality Certified status 
iS. Made, available. to -contracting cfficers for 
consideration in source selection decisions. 


Crm Derr Eney Report Inge oy stem J0DRS) s 9 (DBAS) 


Ars as an aut OMarea Gata base that. trac customer 
complaints on material in the supply system or the 


field. Customer complaints are thoroughly 
investigated, contractor liability determined and 
corrective action initiated. When contractor 


liability is determined on an excessive number of 
customer complaints against a specific item, then 
Quality Assurance informs Procurement that the 
manutacturer 1S a peor risk for future awards. 


Army Contractor History Files: (DBAS) Data bases 
developed separately by individual contracting 


activities in the Army Materiel Command to record 
Gualwiny hiseory- On 1d1 vidual "Contracts Data. Can 
be manipulated to synopsize quality performance of 
epecitie, Contractors. Information forwarded -tc 
contracting officers on request. 


AIR FORCE 


Air Force Systems Command Contractor Performance 
Assessment Report (CPAR): (DBAS) This manual data 


base system provides detailed information and an 
assessment of the on-going performance of selected 
contractors on major systems procurements. The 
data base consists of the narrative assessment of 
the project manager as supported by his functional 
experts, coupled with surveys and questionnaires 
returned by other agencies. Each buying division 
has a library of all CPAR’s. The data is available 
EO. Officials fer CCOonsiderdti0n fin Making ~ new 
contract awards. 


a Force Logistics Command CommeaGec © 
Responsibility Review Program (CRRP): (DBAS/AS) 


This automated data base and application system is 
used to support responsibility determinations for 
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smaller procurements. Inputs include quality and 
delivery data. 


DEFENSE LOGISTICS AGENCY 


Defense Logistics Agency (DLA) Contractor Prentiss. 
System: (DBAS) This proposed data base system will 


contain various historical views of Contra oii 
information from preaward survey data through 
delivery of the end item. This system is broad 
based and will contain data Gm production” 


Galicia. Contract administration, financial 
services, engineering, property, procurement, and 
legal. This system is intended to be an 


information sharing system with the military 
services tied in via the DoD network. 


DLA Standard Automated Materiel Management System 
(SAMMS) : (DBAS) This is the DLA interactive, 


multifunctional data base system. It is comprised 
of the requirements, distribution, technical 
financial and contracting subsystems. Among its 
other Túneles, the Tatter component 
automatically generates purchase requests and 
trailers containing pasi p periformance datar 


DLA Mechanization of Contract Administration 
services (MOCAS): (DBAS) This is the present data 
base system concerning contracts assigned to DLA 
for administration. This system contains much 
current delivery history, financial informaticas 
and other contract/ contractor data; Some Graeme 
military services are already tied in to this 
Varens 


DLA Preaward Survey System (PASS): (DBAS) This 


automated data base system, designed as a subset 
to an overall contractor profile data base system, 
provides information related to a contractor's 
past performance. The PASS includes, but is not 
limited EOF production, quality assurance, 
transportation, property and Linancial/accoun ang 
information arranged in a series of on-line 
preaward survey forms. DLA provides access to 
buying activities for on-line queries. 


DLA Customer/Depot Complaint ystem (CDCI: (DEAS) 


This is a standardized data base system designed 
to track and monitor quality complaints received 
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By the individual supply centers from customers, 
supporting supply activities, and service and DLA 
depots. Types of complaints received include 
ODER’ Ss, (ROD sand storage Guality control reports. 
CDCS interfaces with SAMMS. 


PAOLA Ouality Evaluation Program: (DBAS) The OEP is 


a quality history data base system developed in 
response to IG and GAO findings that DLA was not 
using available Government quality data in 
awarding Come raecls. OEP is currently used 
specifically to aid in an assessment of a 
contractor responsibility. Data is maintained on 
Born a Contractor aña an item basis. OEP includes 
mar ormation on first article testing,  postawara 


orientation conferences, quality systems 
management visits and reviews, product waivers and 
deviations, quality assurance letters ‘one 


instruction, contracts reviewed by the quality 
element, and special quality assurance action 
information. 

These are the primary sources of information for each 
respective service or agency, and there is nothing which 
prevents one service from using the data base of another 
service or DLA. None of the data bases discussed in the 
preceding paragraph were specifically designed for use with a 
particular methodology for conducting past performance 
evaluations in source selections. They all lacked consistency 
in data elements, and because they are maintained for 
different purposes, there is a lack of uniformity in how the 
@etanmis [Ormatted [Ref. 22: p. 9]. Further, none of the data 
bases include performance information for all types of 


contracts and requirements. 


>> 


G. SECONDARY SOURCES OF PERFORMANCE DATA FOR THE PRA 
There are many secondary sources of data to supplement 
these primary sources, and these include, but are not limited : 


to the following [Ref. 24]: 


Ts Certificates of Competencies (COC's): If a small 


business is determined to be non-responsible by a 
Government buying activity, the small business can 
request that the Small Business administration 
determine whether the business is responsible. If 
the SBA determines that the small business is 
responsible, then the SBA will prepare a COC to 
document that determination, and will send a copy 
to NMOAO for inmenso EDAD 


Ze Quality System Reviews: Performed by DCMC, they 


involve an evaluation of the contractor's quality 
procedures and verification that the contractor’s 
quality practices conform to those procedures. 
Navy activities receive copies of quality system 
reviews if they request a copy. 


ou Product Oriented Surveys: These are technical 
product inspections conducted in a conkracreemee 


plant when a buying activity desires to perform a 
special test on an item. They are performed by 
DCMC when requested by the buying activity. Copies 
of them are available upon request from NMQAO. 


oe Discussions with technicians and program managers 
ECOM actual performance experience, where 
available. 


oe Contractor History Files: Data bases, both manuam 
and automated, maintained separately by individual 
DOD contracting activities to “record qua nia 
history on individua contracts. 


Cz Visits to contractor facilities by PRA team 
members. 

ve Receipt Inspection Records 

Be. Production Lot Testing reports 

Ex References cited by offerors in their proposals. 
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10. Telephone interviews or written inquiries with 
Selected Points “ol “Contact. {POCS)...2n “any way 
knowledgeable of a contractor’s previous efforts. 


H: SUMMARY 

This chapter examined performance risk from a legal 
standpoint, described a five step procedure for performing a 
PRA, and listed and described the various primary and 
secondary sources of past performance data available to DoD 
contracting activities today. The next chapter will present 
the methodology used to obtain feedback on the source 
selection procedure model from DoD personnel, and will 
Summarize some of the feedback received and modifications made 


to the model in response to the feedback. 


ay 


V. FEEDBACK AND ANALYSIS 


A. INTRODUCTION 

This chapter will present the methodology used to obtain 
feedback on the source selection model. The feedback was 
obtained from knowledgeable and experienced PCO’s, contracting 
policy officials, contract specialists and instructor Sesa 
contracting, and the DoD/Joint Service/Agency PAT. The 
information received will be presented by activity submitting 
it and by classification of the response. The comments not 
Supporting the model will be analyzed and addressed 
separately. Modifications to the model based on this feedback 


will also be described. 


B. METHODOLOGY 

The refinement of the source selection model presented in 
Chapter III and Appendices C, D and E should be an iterative 
process utilizing revisions based on input fron eebee 
contracting activities as well as commercial purchasing 
departments, and by use of an actual pilot test at a field 
contracting activity. The first step in this refinement 
process consisted of the forwarding of Chapter III and 
Appendices C, D and E along with a cover memorandum to 
knowledgeable and experienced naval officers and senior civil 


service officials working in the contracting “or polis 
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divisions, Cee teaching, InsLructrvon "billetes of cisix 
Significant Navy commands: Naval Air Systems Command (NAVAIR), 
Naval Sea Systems Command (NAVSEA), Military Sealift Command- 
Peetric (MSCP), Naval Supply Center-= Puget Sound, the Office 
of the Assistant Secretary of the Navy (Research, Development 
and Acquisition), and the Naval Postgraduate School, Monterey 
(NPs). In addition, a copy was forwarded to the Navy 
representative on the DoD/Joint Service/Agency PAT, Mr. 
Richard Findley. 

The recipients of the source selection model were 
requested to read the model and provide constructive feedback 
on any aspect of the model in which they may have knowledge 
and experience. It was made clear that all comments regarding 
technical, theoretical, procedural, legal or practical aspects 
were encouraged for submission. All recipients with the 
exception of NSC Puget Sound responded to the feedback 


request. 


C. FEEDBACK AND ANALYSIS 

The responses to the feedback request will be summarized 
by command or group under four different headings based upon 
the category they fell within: comments in favor of the 
general concept, comments not in favor of the general concept, 
comments in favor of specific aspects of the source selection 
model, and comments not in favor of specific aspects of the 


source selection model. If a command did not have a response 
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in one or more of the four categories, it will not apra 
under that category. The comments that weren't in favor of the 
general concept and specific aspects of the source selection ` 
model will be addressed as a group after subsections (2) and 
(4). 
1. Comments in Favor of the General Concept 
NAVAIR [Ref. 25]: 


The incorporation of TOM into the proposal evaluo CoN 
process is an important and worthy goal. 


The model provides an excellent template for others 
interested in incorporating TOM with the evaluation 
process. I think that the model achieves many of its 
objectives. 


The model offers a good starting point, but it must be 
altered to fit the needs and circumstances of the individual 
procurement. 

NAVSEA [Ref. 26]: 
TOM/TOL in the source selection process is workable, and the 
concept is very current and useful. The effort put forth in 
constructing this model is noteworthy. 


ASN(RD&A) [Ref. 28]: 


I believe the technique you have developed along with the 
rating categories are very good. 


NPS [{Ref. 29]: 


I think this“is a great concepit. 
Good ideas and lots of work/thought evident. 


2. Comments That Weren't in Favor of the General Concept 
NAVAIR [Ref. 25]: 


In general, parts of the model are OK- most parts are too 
cumbersome and more trouble than they are worth. 
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NAVSEA [Ref. 26]: 


If the procurement process utilizing this model, or some 
Version of it, iS net performed correctly, contractors may 
try to resort to brochuremanship to make the Government 
happy, the last thing the Government wants. 


ASN(RD&A) [Ref. 28]: 


My concern is not with the approach you are taking, but in 
my professional view, the "over taxing" of our source 
Selection process. Application of TOM into complex source 
selections which already require detailed analysis and 
review by evaluators is not sound. 


To believe that we would pick a contractor based on his TOM 
package on a major source selection is extremely short 
Sighted. 


My real concern is we are compounding the source selection 
process with too many items which are effectively driving us 
Mere “and more away from fully using our "common sense" 
Sapabilities when conducting a source selection. The fact 
that GAO and/or a court upheld our actions doesn’t mean it 
is good business. In source selection, simple is always 
better. 


I regret that I cannot be more receptive to your paper but 
my concerns are "good idea" but by the "wrong" approach 


Source selection evaluations should not be the panacea to 
ensure that TOM is embraced. 


NPS [Ref. 29]: 


Most of the subfactors in Appendix C are too detailed and 
administratively burdensome. 


I guickly came to the conclusion that I want to be the 
iMmeumoent contractor when this SSEB meets. If company "TAT 
has performed this contract previously and has data or 
"evidence" to support its contentions regarding TQM/SPC 
records, then I see it as having a large comparative 
advantage over competitors. 


DoD/Joint Service/Agency PAT [Ref. 30]: 
Your approach of developing evaluation criteria, while 


technically acceptable, may not be viewed favorably by 
Come Gaclors ; 
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The central theme of the comments not in favor of the 
general concept was that the process of incorporating TOM into 
the source selection process was too administratively ` 
burdensome, and that it complicated an already complex system 
to the point where common sense could no longer be effectively 
applied. Other points reflected the thought that the process 
would result in brochuremanship from contractors, and Chaana 
incumbent contractors would have an inherent advantage over 
other competing firms. 

Chapter III pointed out that this model is designed to 
be flexible, and part of its flexibility is that it need not 
be utilized in its entirety. A contracting activity mma eae] 
those subfactors or subordinate subfactors whic PESE E= 
applicable to the particular procurement. In order to make 
the model more streamlined and less administratively 
burdensome without compromising the intent of the procedure, 
nine of the original 23 subordinate subtiactors were deman 
because they overlapped other subordinate subfactors, or were 
either too difficult to evaluate or were inapplicable to DoD 
acquisition: 

The other two points may not be potential sources of 
problems or complications if the model process is conducted 
properly. The PRA is designed to evaluate actual evidence of 
past and current performance independent from the offeror's 
proposal. If the offeror resorts to "brochuremanship" or 


"fluff," the PRA will detect the anomaly, and will reflect the 
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ee Or s truce performance in the overall rating. Incumbents 
should not have an inherent advantage if the contracting 
acta v1ity tallors the source selection process to minimize 
Bras. if firms other than the incumbent do not have 
Significant past experience which can be evaluated by the PRA, 
then the SSEB can assign a "neutral" risk to the offeror on 
the PRA. A "neutral" risk level will not adversely affect the 
overall factor rating. If the incumbent’s performance has been 
emener outstanding" or "marginal" , then this fact will be 
appropriately reflected in the evaluations. 
This model was not designed with the intent of placing 
TOM or other quality factors as the sole or even as the major 
source selection criteria for major systems acquisitions. TOM 
Mou one subtactor among many factors and subfactors. This 
is why the model designed TOM to be subfactors and subordinate 
subfactors, as they are a part of a more encompassing set of 
evaluation criteria. TOM can play as large or as small a role 
in source selection decisions as the contracting activity 
T S. This is the flexibility built into this model. 
3. Comments in Favor of Specific Aspects of the Model 
NAVAIR [Ref. 25]: 
Mes inclusion of a requirement for TOM training for 
evaluators is excellent. This is essential to an effective 
evaluation. 
MSCP [Ref. 27]: 


I like the concept of strong and weak point narratives in 
Appendix C. 
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DoD/Joint Service/Agency [Ref. 30]: 
Your expansion on “Instructions to Offerors"” is Good same 
provide good overall information that goes a long way toward 
ensuring that the offerors don't attempt to provide "MAA 
instead of substantiated data. 


4. Comments That Weren't in Favor of Specific Aspects of 
the Model 


These comments were not taken for action in the 
modification of the model. 

NAVAIR [Ref. 25]: 
An important concern related to the model is the reliance on 
numerical scoring...I would require an adjective rating for 
each listed criteria, preferably with comments/rationale 
from the evaluators to support the adjectives assigned. 
I think that 23 pages of evaluation forms is excessive and 
is likely to become counter-productive as evaluators 
struggle with that much paperwork. It also encourages 
proposals to be excessively lengthy. 

NPS [Ref. 29]: 
Appendix D looks OK- but if everything in Append =. 
translates to Appendix D and evaluation forms, it’s too 
detailed and burdensome. 


DoD/Joint Service/Agency [Ref.30]: 


Be aware that Navy policy has recently been emphasized that 
numerical ratings are not to be used. 


The central theme of the comments that weren’t in 
favor of specific aspects of the model was that numerical 
scoring techniques were discouraged by the Navy, and makes the 
acquisition process vulnerable to protests. The other comments 


reflected that certain areas of the model are cumbersome. 
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The Navy guidance was based on a policy memorandum 
Mete Assistant Secretary of the Navy (Shipbuilding and 
Logistics) that prohibited the assignment of a numeric score 
to the proposed cost or price ina best value decision [Ref. 
31]. The problems this memorandum addresses are situations 
Mee, COMLCracling activities assigned numeric scores to both 
SGce/price and technical evaluation factors to arrive at a 
total point score. The source selection process in the 
researcher’s model does not address cost or technical factors 
other than TQM and quality in general. The model is flexible 
enough to allow for numerical scoring, adjectival ratings or 
narrative comments, or some combination thereof. The model 
used a combination of all three systems to illustrate the 
Mie MoMlity of the procedures. In addition, the policy 
memorandum preserved the right for Navy contracting activities 
Beomuse numerical scoring on non-cost/price factors [Ref. 31]: 

Pemeracting Officers may-contLinue to use numeric scoring 
in evaluating the technical portion of an offeror’s 
proposal. 
D. MODIFICATIONS TO MODEL BASED ON FEEDBACK 
The following modifications have been made to the source 
selection model based upon feedback from the respondents: 
(a)? Every subordinate .subfactor contained the phrases 
"provide credit for" and "evidence of." These phrases were 
replaced by "examples may include but are not limited to" and 


paocumentation/ facts verifying the existence on 
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respectively, because some respondents pointed out that the 
evaluation criteria as written made the Government appear as 
though it were mandating specific TOM practices to: 
contractors. The word "evidence" was not forceful enema 
according to one respondent, who felt that the term "evidence" 
did not adequately describe what the evaluators needed to look 
for when evaluating proposals. 

(2) Under subordinate subfactor A(1), Senior Executive 
Leadership, the example of "the request for documentation of 
senior executive leadership in and communication of quality 
excellence outside the company to various businesses, 
professional and community groups" was deleted because several 
respondents felt as though it was not directly applicable to 
contracting with DoD, and that it invited submissions 
"EON Em 

(3) The “long term" planning mentioned in several 
subordinate subfactors were changed from "3-5 years" to "3-5 
years or more" because a respondent made the case that 3-5 
years was not a long enough time frame to be considered as 
true long term planning. 

(4) sūubordinate subta tor- C2), Quality Leadership 
Indicators in Praning, was deleted because several 
respondents pointed out that it was difficult to verify = mma 
had no real applicability Eo Dob teontracring. 

(5) Several respondents felt that the subordinate 


subfactor, Human Resource Management, of the Human Resource 


66 


Utilization subfactor was already covered by subordinate 
subfactors D(1) through D(4). Therefore, the Human Resource 
Management subordinate subfactor was deleted to avoid the 
double-weighting of this area. 

(6) The subordinate subfactors "Quality Assessment" and 
"Documentation" of the Quality Assurance of Products and 
Services factor were deleted entirely because there was 
Significant overlap between it and the subordinate subfactors 
PAM through & (4). 

i the subordinate subfactor “Complaint Resolution for 
Ouelity Improvement" of the “Customer Satisfaction" subfactor 
was deleted entirely because there was significant overlap 
between it and the subordinate subfactors F(1) and F(2). 

(8) The entire "Past Performance" subfactor, including the 
subordinate Subfactors: Oia laity Or Products/Services 
Delivered, Compliance with Schedule, Integrated Logistics 
Support and Compliance with Cost/Price Parameters, was deleted 
because of many respondents pointing out the significant 
potential for overlap and consequent "double-weighting" 
possibility for the past performance areas. In addition, the 
PRA evaluates past performance almost exclusively, so the 
potential "double-weighting" of past performance between the 


basic proposal evaluation and the PRA was eliminated. 
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E. SUMMARY 

This chapter presented the methodology used to obtain 
feedback on the source selection model from knowledgeable and - 
experienced DoD personnel, and summarized some of the feedback 
received and modifications made to the model in response to 
the feedback. The final chapter will present the conclusions 
and recommendations of the thesis, the answers to the research 


questions, and recommended areas for further research. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. INTRODUCTION 

THIS Chapter will discuss the CONCUSSIONS and 
recommendations resulting from the thesis research. After 
presenting the conclusions and recommendations, the research 
questions will be answered. Finally, recommendations will be 


made concerning areas for further research. 


B. CONCLUSIONS 

The research upon which this thesis is based consisted of 
an extensive literature review, interviews with Government 
contracting, legal and policy personnel, and feedback on the 
source selection model from knowledgeable and experienced 
personnel at Six Navy commands and the DoD/Joint 
Service/Agency Process Action Team. Based upon this research, 
the researcher concludes: 

LE There exists in DoD today many programs, 
regulations, mandates and other incentives to 
PNeCOnDOraLe “TOM OF other quality actors Into tne 
source selection process, Just as commercial 
procurement practices do. 

oe Navy PCO’s are willing to utilize quality factors 
in their source selection process if they could 
have access to a feasible, legal and generally 
accepted method of source selection incorporating 


prestated quality measures, such as TQM. 


3: a OO TA E T pr oduct on NACo eer. hol? -eniid 
generation night vision devices was awarded 


69 


utilizing TQM and SPC in the source selection 
process. ACEC’s source selection process was 
feasible and legal, and only time will tell if the 
source selection process was successful in meeting 
the Army’s goals for quality, timeliness, cost 
control and low costs of ownership. 


The DoD/Joint Service/Agency PAT’s Draft Guide for 
Incorporating TOM in Source Selection was “nee 
finalized, and consequently did not promulgate a 
specific or general model source selection plan 
with evaluation guidelines or criteria. 


The model source selection plan described in 
Chapters III and IV and Appendices C, D and E or 
this thesis is a synthesis of the most feasible 
and viable procedures of the ACEC procurement, the 
PAT draft guidance and ideas of the researcher. 


The model source selection procedures are flexible 
enough to be tailored, expanded or otherwise 
modified by local procurement activities Int oma 
workable source selection method. 


Based upon an unscientific sampling of comments 
from Navy contracting activities, the general 
climate towards acceptance or usage of the source 
selection model is favorable. This favorable 
acceptance is contingent upon the requirement that 
the model be streamlined and less burdensome to 
both Government and contractor, and that the local 
procuring activity be able to tailor or modity fis 
as necessary. 


There exist pockets of strong resistance to the 
incorporation of TQM or other prestated quality 
factors into the source selection process due to a 
variety of philosophies, attitudes and 
perceptions. More universal acceptance of the 
model or the concepts behind the model may be 
gained as personnel in influential policy Seu 
contracting positions become more educated on or 
knowledgeable of TQM or other quality concepts. 


The construction of a solid, viable and feasible 
source selection model which incorporates TQM or 
other quality factors into the DoD source 
selection process is an iterative process 
requiring continuous refinement, feedbacx, 
modifications anal Torres tinos 
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C. RECOMMENDATIONS 

Based upon the research end conclusions, this researcher 
suggests the following recommendations. 

1. The DoD/Joint Service/Agency PAT should continue 
efforts to complete and finalize for distribution, the 
guidance for integrating TQM in the DoD source selection 
process. If or when this is done, the contents of Chapter 
III and Appendix C of this thesis should be considered for 
incorporation into the final draft. 

The successor to the Honorable John A. Betti will have an 
extensive degree of influence over the decision whether to 
continue the effort to finalize the Draft Guide or not. If 
the decision is to continue with the guidance effort, then the 
PAT should consider the contents of Chapter III and Appendix 
C of this thesis. This thesis presents logical extensions, 
amplifications or modifications of the Draft Guide based on 
the literature research, interviews and constructive feedback 
from Navy contracting activities. 

2. Efforts should be made by future researchers to 
further refine this researcher’s source selection 
procedures model. 

As recommended in Section D of this chapter, this 
researcher’s source selection procedures model requires 
further refinement and update based upon input from other 
Government contracting activities, and from the private sector 
procurement personnel. An excellent method to further refine 


it would entail an actual pilot procurement test at an 


activity such as NAC Indianapolis or NWC China Lake. 
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3. Government contracting activities and commercial firms 
should institute or expand their training and education 
programs to include sessions or courses on the 
relationship between TQM or other quality factors and the 
source selection process. 

One of the major impediments to acceptance or utilization 
of source selection procedures which incorporate TQM is 
education and awareness for both Government and commercial 
personnel. Inflexible attitudes towards quality factors and 
TOM are due in large part to fears of the Unkmevny 
misconceptions and resistance to change. Such barriers can be 
removed or lessened most effectively through unbiased, 
objective presentation of the basic fundamentals and 
definitions of quality and TQM as they relate to DoD 
Gomera air Ss. 

4. The Navy should make an attempt to bridge the 
Significant gap between it and the other services and DLA 
in developing and utilizing quality history data bases 
which can incorporate contractor past performance into the 
source selection process. 

In terms of the quantity, quality and maturity of data 
management systems which collect, store and disseminate 
contractor quality history for contractor responsibi orr 
performance risk determinations, the Navy is far behind the 
other services and DLA. This point is reinforced graphically 


by a cursory review of the answer to subsidiary question 


number 5. im section D of this Phapter: 


WZ 


D. ANSWERS TO THE RESEARCH QUESTIONS 


Primary Research Question: What would be a useful and 
practical source selection procedure model for PCO's and 
SSEB’ s which would incorporate significant and 
quantifiable quality factors with a procedure to verify 
and validate offered contractor quality information with 
contractor quality history? 

A useful and practical source selection evaluation 
procedure model would be a synthesis of the ACEC’s method of 
procurement for the third generation night vision devices, 
with the evaluation guidelines or criteria suggested by the 
DoD/Joint Service/Agency PAT in their Draft Guide. The model 
me Ets most basic form consists of the determination of an 
adjective rating and a performance risk level for each 
evaluation subordinate subfactor based upon an evaluation of 
the basic proposal and a Performance Risk Assessment 
respectively. These separate evaluations would be conducted 
by two independent teams within the SSEB, and the basic 
proposal rating and the risk assessment level would be 
combined by the SSEB to form an overall rating for each 
evaluation factor. The overall ratings. plus - supporting 
documentation and narrative comments would be forwarded by the 
3 FE to the SSAC or SSA for a final award decision: 

This model procedure is flexible because numerical 
weighting, adjectival descriptions and narrative comments can 
all be used either alone or in some combination with the 


others. Thesevaluatron criteria. +os the basic proposal are 


modified application preparation guidelines for the Malcolm 


T? 


Baldridge National Quality Award. The application process for 
this prestigious award is generally considered to be a very 
thorough and effective evaluation of a company’s quality >- 
attitudes, values and assurance procedures. The contents of 
the subordinate subfactors can be tailored, or entire 
Subordinate subfactors added or deleted depending on local 
procurement activity regulations or standard operating 
procedures. The PRA can utilize any available contractor 
historical quality data collection system or other sources 
contractor quality data such as described in Section eae. 
Chapter IV. The Army has proven that a similar source 
selection procedure can be defended successfully by GAO and 
the Federal District Courts as long as the contracting 
activity conducts the evaluation process by the same method 


that it describes “in tite solera ne 


Subsidiary Research Question 1: How are Packard Commission 
findings and recommendations, and DoD-wide TOM 
implementation efforts spawning attempts to incorporate 
quality considerations into DoD source selection 
practices? 


After the Packard Commission performed an in-depth study 
of both commercial and Government procurement practices in 
1986, it determined that DoD procurement has a tendency to 
emphasize award to the lowest priced offer at the expense of 
other important objectives such as quality, delivery and low 
life cycle cost. On the other hand, commercial procurement 


officials simultaneously pursue several objectives: attraction 
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pie ne best Qualified suppliers, validation of product 
performance and quality, and the securing of the best price. 

To remedy this situation, one of the Commission's 
recommendations was to revise Federal Law and DoD regulations 
to provide for substantially increased use of commercial-style 
competition, emphasizing quality and established performance 
as well as price. Price should not be the sole determinant, 
especially for procurement of complex systems and services, 
secording to the Commission. 

There are numerous other efforts, programs and regulations 
which have influenced the attitude of DoD procurement 
activities towards quality or TQM as source selection 
Criteria. They range from an Executive Order by the President 
of the United States to the use of TOM and Statistical Process 
Control factors in a best value procurement of a major weapon 
system by the Army Comin a E Command: A 
major effort to develop guidance for DoD in implementing TOM 
throughout not only the defense establishment but also the 
defense industrial base, began with the Honorable John A. 
Betti's formation of a DoD/Joint Service/ Agency Process 
Action Team. This team’s central mission was to develop a 
guide for integrating TQM into the source selection process. 
However, the effort has not progressed beyond a second 
revision of the Draft Guide due to Betti’s resignation. 

The combination of the desire to take the Draft Guide a 


few steps further, and of the success of the ACEC procurement 


To 


in integrating TOM, SPC and a Performance Risk Assessment into 
the source selection process has formed the backbone of the 
model source selection process described in Chapter III of - 


elas Soleo 


Subsidiary Research Question 2: How did the 1984 
Competition in Contracting Act impact PCO use of best or 
qreatest value source selection procedures? 


Since CICA was implemented in 1984, there has been a 
widely held view that the defense procurement emphasis on the 
importance of price has actually intensified. There are three 
reasons for this based according to the Packard Commission: 

1. the universal interpretation of the a rE: 
intentions as being that the Government must buy 
from the lowest bidder; 

A the notion among contracting activities that Crei 
precluded the use of qualification criteria, the 
consideration of technical expertise or of life 
Cycle- costs; and 

oe the resulting focus on the numbers of competitions 
rather than on the success the competition 
achieves in terms of better products. 

The goals and intent of CICA have been obscured by this 
notion that full and open competition precludes best or 
greatest value procurements, and GAO decisions have served 
only to reinforce these interpretations and notions. However, 
the recent passage of the FY 1991 Authorization Act (P.L. 101- 
510) has resulted ina revision of the FAR, and a removal of 


the restriction on award without discussion in instances where 


the award does not result in the lowest overall cost to the 
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Government. This reversal of the impact of CICA on the use of 
best value procedures on DoD acquisition is a positive step 
towards mollifying some of the adverse effects of CICA's 


implementation and interpretation. 


Subsidiary Research Question 3: What are the current 
attitudes of Navy PCO’s, PM’s and commercial procurement 
personnel towards quality factors in DoD source selection? 


CDR Charles A. Perkins, SC, USN performed an extensive 
Survey of personal interviews with Navy PCO’s, Navy Program 
Managers (PM’s) and members of the National Association of 
Purchasing Management (NAPM) -in 1988r -as parts Or. Nis 
dissertation research. Utilizing questionnaires, interviews, 


nonparametric and exploratory data analysis, CDR Perkins 


concluded: 

Hie price competition is a more significant factor in 
Navy procurement than in commercial/industrial 
purchasing; 

a the effectiveness of the Navy procurement process 


im obtaining quality products is inadequate 
because of the over-emphasis on price and the poor 
feedback of accurate and timely quality related 
Mormon rom the na—-users back to Lhe Navy 
BCO; 


Se Bie Comparison of --hevattrteuGdes and prekerences of 
the sample populations of Navy PCO’s and NAPM 
members showed a significant difference in the 
importance that they place on quality factors in 
their purchase decisions, with heavier attention 
placed on it in the commercial/industrial sector. 


a Satisfaction of the minimum specifications drives 
the award in price-based decisions; and 


e considaeration ot other nons=price related factors 


evokes a mot Lon that the item 1s golapleated, or 
that too much is being paid for an item. 


vy 


Subsidiary Research Question 4: What quality values are 
considered as being the most critical and quantifiable to 
Navy PCO's today? 


The results of CDR Perkin's dissertation research also 
confirmed that there is a definite concurrence among Navy 
PCO’s, PM’s and commercial procurement personnel that 
performance and reliability are the numbers one and two 
quality factors respectively in their ranking of the 
importance of quality factors. Navy PCO’s ranked the remaining 
quality factors in order of performance as maintainab i kE 
schedule, durar l'ity, past performance and warranty 
respectively. Interviews with a cross section of Navy PCO’s 
by CDR Perkins also confirmed his hypothesis that a model 
method of source selection based upon prestated quality 
measures would in fact be utilized in some of eeu 
procurements if a feasible and generally acceptable model were 


available to them. 


Subsidiary Research Question 5: What are the various 
systems throughout DoD which collect and disseminate 
historical contractor quality history data, and could be 
utilized in the performance of a PRA? 


The following is a brief listing of the most current data 
bases designed to collect and disseminate historical 
contractor quality history data (pure application systems as 
opposed to data collection bases are not listed): 


NAVY 


Le Navy Product Deficiency Reporting and Evaluation 
system (PDREP) 
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TOF 


ie 


i: 


Lor 
14. 


Loe 


E. 


ARMY 


Ane ota eO Tn ormat on system (015) 


A Contractors Requiring _ Special Attention 
(CRSA) 


Army Commodity Command Standard System (CCSS) 


Army Contractor Performance Certification System 
(SP 7) 


Army Deficiency Reporting System (DRS) 
Ay (oO lmeaoeOnr History Files 


AIR FORCE 


Air Force Systems Command Contractor Performance 
Assessment Report (CPAR) 


ae Foree LoGrstics Command Contractor 
Responsibility Review Program (CRRP) 


DEFENCE, LOGISTICS «AGENCY 
Defense Logistics Agency Contractor Profile System 


DLA Standard Automated Materiel Management System 
(SAMMS) 


Bie Machani zall one Of \ Contract. — Administration 
Services (MOCAS) 


DLA Preaward Survey system (PASS) 
DLA Customer/Depot Complaint System (CDCS) 
Wee Quality Evaluation Program 


AREAS FOR FURTHER RESEARCH 


Take” the, source selection model and send it to 
knowledgeable and experienced purchasing officials 
no ne seommercial sector for their Input“ Modify 
and refine the model as required to make it more 
peines PEor actual (Use? Dy web contracting 
activities: 
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Take the model and persuade a DoD contracting 
activity to use the model in a pilot procurement 
test. Modify and refine the model as necessary 
based on the results of this pilot test. 


Perform a follow on study EPR the Army 
Communications-Electronics Command best value 
procurement of third generation night vision 
devices at a $47M premium, to determine if the 
premium paid was cost effective. 


Compose a glossary of Total Quality Management and 
other quality related terms, so that there will be 
standard definitions and clear terminology for 
both Government and industry to use when 
contracting tor bestrivalietana quealitye 
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APPENDIX A 
MALCOLM BALDRIDGE NATIONAL QUALITY AWARD 
EXAMINATION GUIDELINES- 1990 


© 


Le Leadership 
1.1 Senior Executive leadership 


1.2 Quality Values 
1.3 Management for Quality 
1.4 Public Responsibility 


© 


2. Information and Analysis 


2.1 Scope and Management of Quality Data and Information 
2.2 Analysis of Quality Data and Information 


3.0 Strategic Quality Planning 
3.1 Strategic Quality Planning Process 
3.2 Quality Leadership Indicators in Planning 
e Duality Priorities 


4.0 Human Resource Utilization 


4.1 Human Resource Management 
4.2 Employee Involvement 
aes Quality Education and Training 
4.4 Employee Recognition and Performance Measurement 
4.5 Employee Well-Being and Morale 
5.0 Quality Assurance of Products and Services 
PpPeEbesi gn and Introduction of Cuality Products and services 
em Process and Quality Control 
5.3 Continuous Improvement of Processes, Products and Services 
5.4 Quality Assessment 
9.2 Documentation 
5.6 Quality Assurance, Quality Assessment and Quality 
Improvement of Support Services and Business Processes 
5.7 Quality Assurance, Quality Assessment and Quality 


Improvement of Supplies 


6.0 Quality Results 


Quality or Products and services 

Comparison of OGality Results 

Business Process, Operational and Support Service 
Quality Improvement 

Supplies Quality Improvement 


AADA 
CMN 


OY 
Hs 


ot 


7.0 Customer Satisfaction 


JNN JJ ~) ~) ~) ~] 


0 —J > UN & WN FP 


Knowledge of Customer Requirements and Expectations 
Customer Realtionship Management 

Customer Service Standards 

Commitment to Customers 

Complaint Rsolution for Quality Improvement 
Customer Satisfaction Determination 

Customer Satisfaction Results 

Customer Satisfaction Comparison 
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APPENDIX B 


EXCERPT FROM THE DoD/JOINT SERVICE/AGENCY PAT: 


(DRAFT) GUIDE TO INTEGRATING TOTAL QUALITY MANAGEMENT 


INTO SOURCE SELECTION 


iii. Process for Incorporating Total Quality Management in 


Source Selection 


A. 


Total Quality Management in the Acquisition Plan 
(AP)-The AP should discuss the approach suggested in 
cars “aquide 1 Eo 1 Imcorporarte/ a Wwerrormance Pisk 
evaluation (using Total Quality Management and past 
performance) as part of the source selection. 


Total Quality Management in the Source Selection 
Plana (Sse2)> The5s?2 is <a key document for initiating 
and conducting source selection. It is prepared by 
the program office and approved by the SSA. The SSP 
should provide the details concerning the evaluation 
methodology to be used in assessing performance 
Mistwiwerher NiO Seceaon VvoOr vihis Guide for further 
information on conducting the assessment of 
performance risk). The SSP should also stipulate the 
specialized training requirements of the performance 
risk evaluators. 


LV Total Quality Management in the Request for Proposals 
(RFP) 


This section provides sample language for use in the 


Executive Summary and Sections L and M of the RFP. 


AL EXeCCcutave Summary ot the REP: 


1) The RFP Executive Summary should highlight the 
Government’s intent to consider continuous process 
improvement and past performance as an overall 
performance risk assessment. 


2) Sample RFP Executive Summary Language: 


" The Government intends to consider each 
offeror s apr caton T or Acon Inus process 
more veméent — principles and. past performance: 
Specifically, a performance risk assessment will 
be conducted to assess the effectiveness of the 
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offeror’s continuous process improvement efforts 
and past performance." 


B. Section L- Instructions to Offerors 


1) The following sample instructions to offemene 
are premised on the Malcolm Baldridge Quality Award 
Guidelines. They were developed by tailoring the 
Malcolm Baldridge criteria in scope and detail and 
by adding DoD-oriented language to make the criteria 
more pertinent for use in DoD source selection. This 
tailoring preserves the basic categories and depth 
of coverage needed to distinguish between the levels 
of maturity of continuous process improvement 
efforts of competing offerors and to specifically 
obtain past performance information. Source 
selection personnel should be encouraged to expand 
upon, reduce or otherwise tailor the language as 
appropriate to their specific solicitation nmeecdén 
The result should request only information that will 
support the performance risk assessment, and should 
not require Total Quality Management plans as 
contract deliverables, data items, etc., to be 
developed or submitted. 


2) The Sample RFP Section L Language: 


'" Volume XX- Information for Assessment of 
Performance Risk, Including Continuous Process 
Improvement Efforts and Past Performance 


Note: The use of a separate volume is required to obtain 
proper visibility and to focus on continuous process 
improvement and past performance. This volume may be 
marked "Source Selection Sensitive" and will be 
treated accordingly by the Government. 


1.0 General. The Government intends to consider 
each offeror’s application of continuous process 
improvement principles and past performance. 
Specifically, a performance risk assessment will be 
conducted to assess the effectiveness of the 
Otreror”s cont nues process improvement 
implementation efforts and past performance. 
However, offerors are cautioned that in conducting 
the performance risk assessment, the Government may 
use data provided by the offeror in its proposal and 
data obtained from other sources. 


2.0 Specific Information and Data. The offeror 


shall provide the information below emphasizing 
documented, verifiable evidence of the effective 
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implementation of continuous process improvement 
Errores ana past performance: Actions presently 
planned shall also be included. Information 
Pr roxvidedi should be applicable to the facilities tor 
location where work under the proposed contract will 
be performed. The offeror shall describe the 
apolicatieon Of  =Geneinuols process improvement 
practices, tools and techniques, and shall submit a 
past performance record on relevant contracts. The 
CLberor Sweprocposea. should address the. following 
areas: 


Leadership- Describe how the senior executives 
Createrand sustain a clear and visable quality value 
system along with a supporting continuous process 
improvement management system EG guide all 
activities of the company. 


Information and Analysis- Describe and demonstrate 
the scope, validity, and management of data and 
information that underlie the company’s continuous 
process improvement management system. In 
particular, describe how the company uses data to 
support a prevention-based approach to quality. 


strategic Quality Planning- Describe the company’s 


continuous process improvement priorities and plan 
to achieve them. 


Human Resource Utilization- Describe the company's 
practices to develop and utilize the full potential 
of the work force and to maintain an environment 
ConGguel ve to cul participar tión, (continuous process 
improvement and personal and organizational growth. 
Summarize quantitative and qualitatively 1) 
accomplishments to date and recent trends in 
employee pave lCipac lon ao continuous process 
im ororen activity, 2) types of continuous process 
improvement education and training provided in each 
pertinent employment category, and 3) trends in 
recognizing employees noe GOner burl ons Lo 
continuous process improvement. 


Quality Assurance of Products and Services- 


Describe how products and services are continuously 
improved through optimization and improvement of 
processes. In the area of design and development, 
where applicable, the description may include 
INPGemMal lon pertaining, tO che usé of «continwvous 
process improvement techniques such as Quality 
Function Deployment, producibility engineering and 
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planning, design of experiments, DoD Directive 
4245.7M -— Transition from Development to Production, 
etc. Include a description of how the offeror flows 
the company’s continuous process improvement focus 
down to subcontractor levels. 


Results - Provide data that shows trends in: 
improvement of quality of products and services 
based on analysis of customer requirements, analysis 
of Quai) deficiency reports, cycle time 
reductions, Material Review Board actions, scrap and 
rework, etc., and the analysis of internal business 
operations, and improvement in the quality of 
Supplies and services furnished by other companies. 
Provide evidence of the use of results to overcome 
and prevent the recurrence of problems. Demonstrate 
application of the offeror’s continuous process 
improvement activities by briefly summarizing 
several projects that illustrate their breadth and 
effectiveness. The offeror should submit 
information on contracts considered relevant in 
demonstrating ability to perform on the proposed 
effort. This information may include data on efron 
performed by other divisions, corporate management, 
critical subcontractors, or teaming subcontragwan es 
1f such resources will be brought to bear or 
Significantly influence the performance of the 
proposed effort. For each current or past (within 
the prior three years) contract which is relevant 
(similar ettort mor for items of comparable 
complexity), the offeror should provide the 
following inr orma ron: 


a) Administrative Data 


(i) Company/Division Name 

(EL) Program Title 

(La Contracting Agency 

(1v) Contract Number and Award Date 

(v) A Brief Description of the Contr ki 


Bi mena, Indicating Whether it Was 
Development and/or Production 

(vi) Type ofm Contract 

CL Period of Performance 

(viii) Original Contract Dollar value and 
Current Dollar Value 


(1x) Original and Current Schedules and 
Completion Dates 
(x) Name, Address, and Telephone Number of 


Government Program Director/Manager, 
Administrative Contracting Officer, and 
PESA 


86 


b) Peci Content  “Orlrerors are. required to 
explain, aspects Of? Ene contracts that. are deemed 
relevant to the proposed effort. Offerors are also 
PeQquesLed. Eos isupbmit. informations on- Significant 
achievements or explain past problems they consider 
pe levant to “the proposed elfort: 


Cuc coner Satis action Describe “the company’ s 
Metneds;( 420m? determining ocustomer- -satisfac ion. 
Briefly summarize the company’s customer 
satisfaction trends." 


Cy Section M — Evaluation Factors for Award 


ie Lee slanguagewror Section M shoulda be Covered in 
two Parts: 


a) General Basis for Award - This section 
describes how the solicited proposals will 
be evaluated and rated. 


BD) FPertormence rrek = Ins ¿section deseribes 
how performance risk will be assessed. 


2) Sample RFP Performance Risk Language: 


The Government will also conduct a performance risk 
assessment based upon: 1) effectiveness of the 
offeror’s continuous process improvements effort and 
the applicability of the offeror’s use on continuous 
process improvement practices, EOS, and 
techniques, and 2) the offeror's past performance 
record, demonstrated in terms of actual results. In 
assessing performance risk, the Government may use 
information and data as submitted by the offeror, as 
well as information and data obtained from other 
sources, to evaluate the areas listed above. 
Performance risk assessment relates to the offeror’s 
Prova lity “cor successfully accomplishing the 
proposed effort." 
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APPENDIX C 
EVALUATION FORM 


FACTOR: 
SUBFACTOR: Leadership 


SUBORDINATE SUBFACTOR: A (1) Senior Executive Leadership 


Senior level management’s active leadership in, personal in- 
volvement with, and company-wide visibility in developing and maintaining 
a favorable climate for quality values. 

Rate the appropriateness, === === 
effectiveness and extent of| Maximum Assessment Score 
executive level involvement Points 
Examples may include but 
are not limited to: 
Documents/facts verifying 
the existence of-- 
- of a Senior Executive 
plan for incorp rating 
the quality values into the 
leadership process of the 
company. 
- a Senior Executive 
system for communication 
to and access w1th 
the company’s personnel.Ex 
amples include: 

* Town Meetings 

* CEO Bulletins 

* CEO Memorandums 

* Company Magazines 





Seering Justifica iont 


IL Strong Points. 


2. Weak Points. 


3. Overall Rationale for Assessment %. 
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EVALUATION FORM 


FACTOR: 
SUBFACTOR: Leadership 


SUBORDINATE SUBFACTOR: A (2) Management for Quality 
Evaluate how the company integrates its quality values into day-to- 

day management of all units. Examples may include but are not limited to: 
= Key strategies for invol= ¡ 
ving all levels of manage- |, Maximum 
ment and supervision in thej, Points 
monitoring and promotion of 
apality, and the principal 
roles and responsibilities 
defined at each level. 
- Key strategies to promote 
cooperation and synergism 
among managers and supervi-| 
sors of all units at all 
levels. Examples include: 

* interunit teams 

* process action teams]|i 
Types, frequency and con- 
tent of company reviews of 
the status of quality plans 
and actions to assist units 
not performing up to poten- 
tial. 





peoring Justification: 


i. Strong Points. 


2. Weak Points. 


3. Overall Rationale for Assessment %. 
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EVALUATION FORM 


FACTOR: 
SUBFACTOR: Information and Analysis 


SUBORDINATE SUBFACTOR: B (1) Scope and Management of Quality Data and 
ITatorma ion. 

Evaluate the adequacy of the company’s management information 
system to support the planning, management and evaluation of qual-ity, and 
how data and information reliability, timeliness and access are assured. 
Examples may include but are not limited to: 

Documents/facts verifying the existence of-- 

= Criteria or 
selection of specific items Assessment Score 
for inclusion in the qual- Points 

ity related data and infor- a 
ation base. | 
- A valid and effective use 
of the data, including int- 
ernal operations and proc- 
esses, quality results, su- 
pplier quality, etc. 

- Processes and 
technologies the company 
uses to ensure the validity 
consistency, standardization 
review,update and timely 
access to quality data 
throughout the company. 





SEOLIN AUS ta ELSA LION 


lo Strong Points: 


2. Weak Points. 


3. Overall Rationale for Assessment %. 
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EVALUATION FORM 


FACTOR: 
SUBFACTOR: Information and Analysis 


HUBORDINATE SUBFACTOR: B (2) Analysis of Quality Data and Information 
Evaluate how data and information are systematically collected and 
analyzed to support the company’s quality objectives 
in a timely and efficient manner. Examples may include but are not limited 
EO: 
Documents/facts verifying the existence of-- - 
Principal types of anal- 
ysis performed such as peel Maximum Assessment 
ermination of trends and Points 
projections of quality imp- 
movements that should res- 
ult from changes in techno- 
logy, evaluation of system 
performance, and assessment 
SE product long-term perfor 
mance. 
= Plans made 
ems teps taken to shorten Il 
the cycle of data gathering 
analysis,and access to imp- 
Beve Support of company 
quality objectives. 
- Analysis perfromed 
leads to changes in types 
of data collected, improved 
Reliability of data,and im- 
proved analytical capabil- 
ities. 





peering Justification: 


strong Points. 


Z. Weak Points. 


BR Overall Rationale for Assessment %. 
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EVALUATION FORM 


FACTOR: 


SUBFACTOR: Strategic Quality Planning 


SUBORDINATE SUBFACTOR: C (1) Strategic Quality Planning Process 
Evaluate the company's strategic quality planning process for 
short term (1-2 years) and long term (3-5 years) to achieve and 
sustain leadership and customer satisfaction 
Examples may include but are not limited to: Documents/facts verifying the 
existence of-- 


- Employees, 
suppliers and customers con|| Maximum Assessment 
tributing to the planning Points 


Process, including data eos e 
llection and analysis. | 
- The company attempts to 
integrate short and long 
term priorities into the co 
mpany’s leadership object- 
ives. 

- There are mechanisms by 
which the company ensures 
that its suppliers meet or 
exceed its quality require- 
ments. 

- The company has attempted 
to perform a self-diagnosis 
or self assessment of 

its quality program such as 
through use of the Malcolm 
Baldridge National Quality 
Award application guideline 
as evaluation criteria. 

- The planning 

process is Continuously evaluated and improved. 





[Coming JUStitiCataon : 


l- Serong Points. 


2. Weak Points. 


3. Overall Rationale for Assessment %. 
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EVALUATION FORM 


FACTOR: 
SUBFACTOR: Human Resource Utilization 


SUBORDINATE SUBFACTOR: D (1) Employee Involvement 

Evaluate how the company provides a means for all of its employees to 
effectively contribute to the company’s quality objectives. Examples 
include but are not limited to: Documents/ 
facts verifying the existence of-- 
- Company encouragement 
Se Group participation | Maximum 
to evaluate and Points 
assist in redesign of the 
company’s systems and proc- 
esses, examples may include 

* Steering groups 

* Improvement teams 

SE roblem=solving 

teams 

meOuality circles 
-Other avenues for employee 
participation, ineluding 
suggestion systems, ombud- 
smen, or hotlines. 
- The company attempts to 
enhance employee power to 
act (empowerment). 
- The company 
utilizes surveys, question- 
naires or other methods to 
collect information and to'i 
provide a baseline measure- 
ment of employee attitudes and values. 
- The company provides prompt, constructive feedback on and rewards for 
employee suggestion, comments and criticisms. 





Scoring Justification: 


iP soeroeng Points. 


2. Weak Points. 


3. Overall Rationale for Assessment 3%. 


2 


EVALUATION FORM 


FACTOR: 
SUBFACTOR: Human Resource Utilization 


SUBORDINATE SUBFACTOR: D (2) Quality Education and Training 

Evaluate how the company provides quality related education and 
training for employees,and how it subsequently utilizes the 
knowledge and skills acquired in the process. Evaluate how the company 
measures human resource contributions so that management can make informed 
decisions about training, recognition and employee involvement. Examples 
may include but are not limited to: 
Documents/facts verifying the existence of-- 


- Company rationale == =M = 
for deciding what education]| Maximum Assessment Score 
and training is needed Points 


for different categories 
of employees. 

- The company 

provides on-the-job reintors 
rcement of the knowledge 
and skills acquired in edu- 
Cation and training. 

=- Indicators of effective: 
ness of the company’s edu- 
cation and training activ- 
ities and how the indicat- 
ors are used to improve 
these activities. 

- Company ties needs-based 
training with business 
plans. 

- Company designs 
results/benefits measure- 
ment into the training proc]! 
ess. 





Scor ind Justification: 
Ieo Cr ond POINCS: 


2. Weak Points. 


00 


3. Overall Rationale for Assessment 
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EVALUATION FORM 


FACTOR: 


SUBFACTOR: Human Resource Utilization 


SUBORDINATE SUBFACTOR: D (3) Employee Recognition and Performance 
Measurement 

Evaluate how the company’s employee recognition and performance 
measurement process supports quality improvement. 
Examples may include but are not limited to: 
Documents/facts verifying the existence of-- 
- Company strategies for en|| Maximum Assessment Score 
Souraging contributions to Points 
euality, including recogn1-5 (== 
pon Of individuals or 
a oups, and suppliers for 
exemplifying quality. 
- The company 
evaluates the effectiveness 
of its recognition and per- 
formance measurement system 
including mechanisms to 
solicit employee feedback. 
Ine company provides 
prompt, constructive feed- 
back on and rewards for 
employee suggestions, 
comments, criticisms. 





meoring Justification: 


E eL ong Points. 


2. Weak Points. 


o 


3. Overall Rationale for Assessment %. 


29 


EVALUATION FORM 


FACTOR: 
SUBFACTOR: Human Resource Utilization 


SUBORDINATE SUBFACTOR: D (4) Employee Well-Being and Morale 
Evaluate how the company safeguards the health and safety of 
employees, ensures their comfort and physical protection, and maintains a 
Supportive work environment. Examples may include but are not limnited to: 
Documentation/facts verifying the existence of-- 
- The company == = 
integrates well being and Maximum Assessment Deore 
morale factors such as Points 
health, safety, job satis- 
faction and ergonomics into 
quality improvement activ- | 
ities. Examples may include 
vie rPiness classes 0 
* Special health eventsll 
* Health risk assess- 
ments for employees 
and their families 
= The quality and ettectiv- 
eness of the company’s meth 
ods’ for analysis of under: 
lying causes of accidents, 
work related health prob- 
lems and job dissatisfact- 
ion, and the elimination of 
adverse conditions. 
- Special services, facil- 
ities and opportunities thel 
company makes available to 
Support employees. 





Scoring Jú tirica on. 


Te Strong Points. 


2. Weak Points. 


3. Overall Rationale for Assessment $. 


J0 


EVALUATION FORM 


FACTOR: 
SUBFACTOR: Quality Assurance of Products and Services 


SUBORDINATE SUBFACTOR: E (1) Design and Introduction of Quality 
Products and Services 

Evaluate the process by which new or improved products and services 
are designed and introduced to meet or exceed customer requirements, and 
how processes are designed to deliver according to the requirements. 
Examples may include but are not limited to: 
Documentation/facts verifying the existence of: 
-eeecective mechanisms for f ; 0 
conversion of customer Maximum Assessment Score 
needs and expectations into Points 
product and process require 
ments and/or service qual- 
ity standards. 
= The company’s methods for 
assuring that quality is de 
Signed-in during research, 
development, and validation 
Stages, including review of 
designs for feasibility,and 
assessment of key factors 
in production and use. 
= The company 
has a detailed control plan 
we ding provisions for 
the selection and setting 
of key products character- 
istics to be controlled and 
how they are controlled. 





Coring Justification: 


OC ong Points. 


2. Weak Points. 


o 


3. Overall Rationale for Assessment 3%. 


9T 


EVALUATION FORM 


FACTOR: 
SUBFACTOR: Quality Assurance of Products and Services 


SUBORDINATE SUBFACTOR: E (2) Process and Quality Control 

Evaluate the processes by which the company’s products or services 
are controlled, and how the company assures that products or services meet 
design plans, specifications or SOW’s. Examples 
may include but are not limited to: Decumentation/tacts verifying 
the existence of-- 


- Effective approaches the (F 
company uses to ensure that]]/ Maximum Assessment 
processes which produce pro Points 


ducts or services are adequll 
ately controlled to ensure 
quality, timely delivery 
within cost objectives. 

- Effective approaches the 
company takes to ensure 
that products and services 
meet design plans, specs or 
SOW’s. 

- Effective approaches the 
company uses to identify 
root causes of process up- 
sets (problems). 

- Effective approaches to 
the design of the measures 
to correct process upsets, 
and methods of verifying 
that the measures produce 
the predicted results, and 
are effectively utilized in 
all appropriate units of 
the company. Í 

- Effective approaches the company uses to utilize the information 
obtained from process and quality control for problem prevention and 
quality improvement. 





Scoring Wustilication - 


1. SStinong Poines. 
22 Weak Point's’. 


3. Overall Rationale for Assessment +. 
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EVALUATION FORM 


FACTOR: 


SUBFACTOR: Quality Assurance of Products and Services 


SUBORDINATE SUBFACTOR: E (3) Continuous Process Improvement of Processes, 
Peoducts and Services 

Evaluate how the company’s products/services are continuously 
improved through optimization and improvement processes. Examples 
may include but are not limited to: 
Documentation/facts verifying the existence of-- 
= ane Company 
utilizes methods for the Maximum Assessment Score 
continuous improvement of Points 
quality in products/service 
using any of the following 
techniques (or any others 
which the offeror presents) 
* Cause and Effect Analysis 
Ra eCo Charts 
pe Seatistical Process Cont- 
rol (SPC) 
Histograms 
Checksheets 
Input/Output Analysis 
Sctatter Diagrams 
Concurrent Engineering 
Control Charts 
Work Flow Analysis 
Shewart Cycles 
= The company 
has a method of integrating 
continuous process improve- 
ment into daily operations 
and routine process and 
quality control by all un- 
mes Of the company. 


+ XX + © + »* 





Coring Justification: 


MES C rong Points. 


2. Weak Points. 


o 


3. Overall Rationale for Assessment %. 


J9 


EVALUATION FORM 


FACTOR: 


SUBFACTOR: Quality Assurance of Products and Services 


SUBORDINATE SUBFACTOR: E (4) Quality Assurance, Quality Assessment and 
Quality Improvement of Suppliers 

Evaluate how the quality of materials, components and services 
furnished by other businesses is assured and continuously improved. 
Examples may include but are not limited to: 
Documentation/facts verifying the existence of-- 
- Effective processes used =) Ia 
to ensure that the company Maximum ASS SS Ie IE 
quality requirements are Points 
being met by suppliers, — 
through such means as: 
audits, inspections, cert- 
ification and testing, part 
nerships, training, cont- 
ract incentives and recog- 
Nilo 





Scoring Justification: 


LASER EOS FOIN 3. 


2. Weak Points. 


3. Overall Rationale for Assessment %. 


100 


EVALUATION FORM 


FACTOR: 
SUBFACTOR: Customer Satisfaction 


SUBORDINATE SUBFACTOR: F (1) Knowledge of Customer Requirements and 
Expectations 

Evaluate how the company effectively manages its relationships with 
customers, broken out by commercial/Government categories, and how it 
ensures continuous improvements of customer relationship management. 
Examples may include but are not limited to: 
Documentation/facts supporting the existence of-- 


- Effective means for ens- 
uring easy access for cust-| Maximum Assessment Score 
omers to comment, seek ass- Points 


istance or to complain. 

> Company follow-ups 

with customers on comments, 
complaints, requests for 
assistance. 

- Mechanisms in 

place to empower customer- 
contact employees to take 
extraordinary measures to 
accommodate the customer 
when necessary. 

- The use of technology and] 
og st1cs (infrastructure) 
support to enable customer-|| 
contact employees to prov- 
ide effective and timely 
customer service. 

- Company analysis of 
complaint information, gains 
and losses of customers,and 
lost orders to assess costs 
and market consequences for policy review. 





oring Justification: 


im Strong Points. 


2. Weak Points. 


3. Overall Rationale for Assessment %. 


TOT 


EVALUATION FORM 


FACTOR: 
SUBFACTOR: Customer Satisfaction 


SUBORDINATE SUBFACTOR: F (2) Customer Service Standards 

Evaluate the company’s standards governing the direct contact 
between employees and customers, and how these standards are set and 
modified. Examples may include but are not limited to: 
Documentation/facts supporting the existence of-- 
- The existence of product 


and service guarantees and Maximum Assessment Score 
product warranties, and Points 


other types of commitments 
the company makes to pro- 
mote trust and confidence 
in its products and serv- 
ices. Evaluate the compre- 
hensiveness, conditional 
qualifications, understand- 
ability and credibility. 

- How improvements in the 
company’s products/services 
over the past 3 years have 
been translated into change 
in warranties, guarantee’s 
other commitments. 





Scöring Ustica rion: 


Io Strong Ports: 


2. Weak Points. 


3. Overall Rationale for Assessment %. 


OZ 


APPENDIX D 
EVALUATION FORM 
(EXAMPLE) 
Basic Proposal Evaluation 
zL COIP Ora rOn- Proposal 


FACTOR: Continuous Process Control 
SUBFACTOR: Quality Assurance of Products and Services 


SUBORDINATE SUBFACTOR: E (3) Continuous Process Improvement of processes, 
Products and Services 

Evaluate how the company’ S products/services are continuously 
improved through optimization and improvement ic <a Examples: 
- Evidence that the company 
utilizes methods for the Maximum Assessment Score 
coninuous improvement of Points 
quality in products/service 
using any of the following 
techniques: 100 
* Cause and Effect Analysisi 
PB earero Charts 
* Statistical Process Cont- 
OL (SEC) 
Histograms 
Checksheets 
Input/Output Analysis 
Scatter Diagrams 
Concurrent Engineering 
Control Charts 
Work Flow Analysis 
Shewart Cycles 
- Evidence that the company 
has a method of integrating 
continuous process improve- 
ment into daily operations 
and routine process and 
aality control by all un- 
its of the company. 
-oring Justification: 
1. Strong Points. XYZ Corporation’s proposal is very detailed and specific 
with regard to its successful use of SPC to control the widget production 
process. Its description of the Measurement Error 
Sevay,Control Charting, Cpk Capability Index, Taguchi Loss Function 
control processes, and SPC applications to assemblies, subassem-blies and 
components show positive trends in continuous process 
improvements as evidenced by a 4% drop in scrap rate last year. 
ANMeak Points. Not all of XYZ Corporation's divisions, affiliates and 
suppliers have this successful SPC and continuous process improvement 
program in place, or they are in place, but not as mature or successful as 
they are for the widget production process. 
3. Overall Rationale for Assessment $: This is clearly a "superior" 
component of the subfactor, but not 100% due to the weak point. 





+ + + E + 
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EVALUATION FORM 
(EXAMPLE) 
Peformance Risk Assessment 
XYZ Corporation 

FACTOR: Continuous Process Control 
SUBFACTOR: Quality Assurance of Products and Services 
SUBORDINATE SUBFACTOR: E (3) Continuous Process Improvement of processes, 
Products and Services 

Evaluate how the company’s products/services are continuously 
improved through optimization and improvement processes. Examples: = 
Evidence that the company TIO 
utilizes methods for the Maximum Assessment 
coninuous improvement of Points 
quality in products/service 
using any of the following 
techniques: 
* Cause and Effect Analysis 
x Pareto Charts 
* Statistical Process Cont- 
rol (SRC) 
Histograms 
Checksheets 
Input/Output Analysis 
Scatter Diagrams 
Concurrent Engineering 
Contro Charts 
Work Flow Analysis 
Shewart Cycles 
- Evidence that the company 
has a method of integrating 
continuous process improve- 
ment into daily operations 
and routine process and 
quality control by all un- 
its of the company. 
Scoring Justification: 
l. Strong Points: A review of PDREP database, preaward surveys and on-site 
plant visits performed in conjunction with two Navy production contras 
within the past two years for a wodget, similar in most respects to a 
widget, revealed that XYZ’s SPC and continuous improvement processes were 
adequate, with some weaknesses noted. 
2. Weak Points: A review of the applicable data in the PDREP data base, 
preaward surveys and results of on-site plant visits reveals notable 
deficiencies in XYZ Corporation's SPC procedures, espec-ally with regard 
to the machined parts provided by a supplier, and 
in XYZ's manufacture of Vacuum Tube Envelopes and microchannel plates. 
Further, it was determined that XYZ was not making any significant effort 
to incentivize its supplier to use SPC in its process, nor was it making 
an effort to locate a better supplier. 
3. Overall rationale for Assessment %: In view of how critical the 
machined parts, Vacuum tube envelopes and microchannel plates are in the 
proper functioning of widgets, and how little progress XYZ has made to 
correct, the risk in this area is high-moderate. 
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APPENDIX E 


Evaluator 
Date: 
REP: 
CONSOLIDATED TALLY SHEET 
(EXAMPLE) 
Basic Proposal Evaluation 
Max QEtTertorÉSs 
Factors/Subfactors Points 
2 3 4 
Leadership and Personnel Management 
A. Leadership 
(1) Senior Executive Leadership Ze 
(2) Management For Quality 30 


B. Information and Analysis 
(1) Scope and Management of Data 20 


(2) Analysis of Quality Data 30 
E Strategic Quality Planning 
(De trategic Quality Planning 100 
Process 
D. Human Resource Utilization 
(1) Employee Involvement 70 
2 Ouality Education ana 
Training TO 
(3) Employee Recognition and 
Performance Measurement 60 
Factor Totals 400 


Continuous Process Improvement 
E. Quality Assurance of Products 


and Services 


(Design and Introduction of 100 
Quality Products/Services 
(MANE rocess and Quality Control 100 


(3) Continuous Process Improvement 100 
of Processes, Products/Services 
(4) Quality Assurance,Quality Ass- 100 
essments and Quality Improve-Suppliers 
F. Customer Satisfaction 
(1) Knowledge of Customer Require- 





ments and Expectations 100 
(2) Customer Service Standards 100 
Factor Totals 600 
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Evaluator 
Date: 
(EXAMPLE) RFP: 
Consolidated Tally Sheet 
Performance Risk Assessment 


Max Offerors 
Factors/Subfactors Points 
2 3 4 
Leadership and Personnel Management 
A. Leadership 
(1) Senior Executive Leadership 20 
(2) Management For Quality 30 
B. Information and Analysis 
(1) Scope and Management of Data 20 





(2) Analysis of Quality Data 30 
C. Strategic Quality Planning 
(1) Strategic Quality Planning WOO 
Process 
D. Human Resource Utilization 
(1) Employee Involvement 70 
(2) Quality Education and 
Training 70 
(3) Employee Recognition and 
Performance Measurement 60 
Factor Totals 400 


Continuous Process Improvement 
E. Quality Assurance of Products 


and Services 


(1) Destgarana Introduction of 100 
Quality Prođucts/Services 
(2) Process and Quality Control 100 


(3) Continuous Process Improvement 100 
of Processes, Products/Services 
(4) Quality Assurance,Quality Ass- 100 
essments and Quality Improve-Suppliers 
F. Customer Satisfactioi 
(1) Knowledge of Customer Require- 





ments and Expectations 100 
(2) Customer Service Standards 100 
Factor Totals 600 420 


TOG 


AN EXAMPLE OF SSEB CONVERSION OF SCORES TO ADJECTIVAL RATINGS/RISKS: 
A. Example Conversion Scales 


Basic Proposal Evaluation Scale: PRA Risk Level Scale: 












Risk 

Factor: Point Range: Rating: Point Range: Level: 
continuous 540-600 Superior 910=600 Low 
Process 420-539 Acceptable 420-509 Moderate 
Improvement 360-419 Marginal 0-419 High 

= 359 )“Unsat 
Leadership and 360-400 Superior 340-400 Low 
Personnel 280-359 Acceptable ZE0=339 Moderate 
Management 240-279 Marginal 0-279 High 

0=239 “Unsat 





em e anple Conversion of XIZ Corporation's Scores by SSEB: 


Risk 
Macror: Points: Rae ing Points: Level: 
CPI 540 Superior 420 Moderate 
LPM 360 Superior 250 High 





C. An example determination of Overall Factor Rating by SSEB: 


Basic Performance 
Evaluation Risk Overall 
Bactor: Rating: Assessment: Rating: 
CPI Süperior Moderate Acceptable 
LPM Superior High Marginal 
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